2023-12-11T19:54:08.886 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-11T19:54:08.886 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265 --owner scheduled_yuriw@teuthology --name yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --verbose -- /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/orig.config.yaml 2023-12-11T19:54:08.926 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-11T19:54:09.008 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265 branch: wip-yuri3-testing-2023-12-07-0727 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7487265' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri3-testing-2023-12-07-0727 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-12-07-0727 sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9380 sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2023-12-07-0727 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMOJn5q+AoDi076ehgEb2UB7o6YSW8kCF1AYQaNGSo6OsPXHsWuu+qyowrTGgX9c9Y5SWcZZUcD7KSBtoPIZFXI= smithi133.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvBdaZakCDp41o3yJ8GUZkZbVVpPi2BSMpZMLhkk3GfTViBdqH1gJ1b4iArrB2Z2NsODYFNDvs6n8UV3I9Ceps= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-11_16:41:51 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-11T19:54:09.008 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa; will attempt to use it 2023-12-11T19:54:09.009 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks 2023-12-11T19:54:09.009 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-11T19:54:09.010 INFO:teuthology.task.internal:Checking packages... 2023-12-11T19:54:09.034 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe' 2023-12-11T19:54:09.034 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-11T19:54:09.034 INFO:teuthology.packaging:ref: None 2023-12-11T19:54:09.034 INFO:teuthology.packaging:tag: None 2023-12-11T19:54:09.035 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-12-07-0727 2023-12-11T19:54:09.035 INFO:teuthology.packaging:sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-11T19:54:09.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri3-testing-2023-12-07-0727 2023-12-11T19:54:09.255 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-83.gc56ad3ea 2023-12-11T19:54:09.256 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-11T19:54:09.262 INFO:teuthology.task.internal:no buildpackages task found 2023-12-11T19:54:09.262 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-11T19:54:09.311 INFO:teuthology.task.internal:Saving configuration 2023-12-11T19:54:09.326 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-11T19:54:09.334 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-11T19:54:09.360 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-11 19:49:01.164038', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMOJn5q+AoDi076ehgEb2UB7o6YSW8kCF1AYQaNGSo6OsPXHsWuu+qyowrTGgX9c9Y5SWcZZUcD7KSBtoPIZFXI='} 2023-12-11T19:54:09.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-11 19:49:01.165693', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvBdaZakCDp41o3yJ8GUZkZbVVpPi2BSMpZMLhkk3GfTViBdqH1gJ1b4iArrB2Z2NsODYFNDvs6n8UV3I9Ceps='} 2023-12-11T19:54:09.380 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-11T19:54:09.388 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-11T19:54:09.388 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-11T19:54:09.388 INFO:teuthology.run_tasks:Running task console_log... 2023-12-11T19:54:09.459 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f401f060e50>, signals=[15]) 2023-12-11T19:54:09.460 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-11T19:54:09.467 INFO:teuthology.task.internal:Opening connections... 2023-12-11T19:54:09.467 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T19:54:09.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T19:54:09.554 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T19:54:09.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T19:54:09.641 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-11T19:54:09.648 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2023-12-11T19:54:09.666 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2023-12-11T19:54:09.667 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2023-12-11T19:54:09.730 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2023-12-11T19:54:09.730 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2023-12-11T19:54:09.730 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-11T19:54:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-11T19:54:09.732 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2023-12-11T19:54:09.766 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2023-12-11T19:54:09.785 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2023-12-11T19:54:09.785 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2023-12-11T19:54:09.843 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream" 2023-12-11T19:54:09.843 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="9" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="9" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el9" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:LOGO="fedora-logo-icon" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-11T19:54:09.844 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-11T19:54:09.845 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2023-12-11T19:54:09.867 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-11T19:54:09.939 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-11T19:54:09.956 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-11T19:54:09.957 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2023-12-11T19:54:09.960 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2023-12-11T19:54:09.977 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-11T19:54:09.983 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-11T19:54:09.983 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2023-12-11T19:54:10.020 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2023-12-11T19:54:10.126 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-11T19:54:10.185 INFO:teuthology.run_tasks:Running task kernel... 2023-12-11T19:54:10.205 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-11T19:54:10.205 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-11T19:54:10.205 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-11T19:54:10.206 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-11T19:54:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T19:54:10.222 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-12-11T19:54:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T19:54:10.223 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-12-11T19:54:10.280 INFO:teuthology.orchestra.run.smithi133.stdout:5.14.0-361.el9.x86_64 2023-12-11T19:54:10.280 INFO:teuthology.task.kernel:Running kernel on smithi133: 5.14.0-361.el9.x86_64 2023-12-11T19:54:10.280 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum install -y kernel 2023-12-11T19:54:10.281 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-361.el9.x86_64 2023-12-11T19:54:10.281 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-361.el9.x86_64 2023-12-11T19:54:10.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2023-12-11T19:54:12.255 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-11T19:54:12.324 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-11T19:54:13.572 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - BaseOS 7.9 MB/s | 2.0 MB 00:00 2023-12-11T19:54:13.925 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2023-12-11T19:54:14.724 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-12-11T19:54:15.296 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 7.9 MB 00:01 2023-12-11T19:54:18.017 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 18 MB 00:02 2023-12-11T19:54:18.830 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 18 MB 00:02 2023-12-11T19:54:22.401 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 5.8 MB 00:00 2023-12-11T19:54:23.993 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-12-11T19:54:24.081 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.8 MB 00:01 2023-12-11T19:54:25.061 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 21 MB/s | 20 MB 00:00 2023-12-11T19:54:25.656 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-12-11T19:54:26.716 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 21 MB/s | 20 MB 00:00 2023-12-11T19:54:29.795 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-12-11T19:54:31.507 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2023-12-11T19:54:32.060 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-11T19:54:32.060 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-11T19:54:32.159 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T19:54:32.164 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2023-12-11T19:54:32.165 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T19:54:32.166 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 96 M 2023-12-11T19:54:32.166 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 121 M 2023-12-11T19:54:32.166 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T19:54:33.764 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-11T19:54:33.764 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-11T19:54:33.848 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T19:54:33.857 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T19:54:33.857 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout:Install 4 Packages 2023-12-11T19:54:33.858 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T19:54:33.859 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 96 M 2023-12-11T19:54:33.859 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 121 M 2023-12-11T19:54:33.859 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T19:54:34.221 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 2.9 MB/s | 5.6 MB 00:01 2023-12-11T19:54:34.325 INFO:teuthology.orchestra.run.smithi133.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 17 MB/s | 5.6 MB 00:00 2023-12-11T19:54:34.817 INFO:teuthology.orchestra.run.smithi133.stdout:(2/4): kernel-core-5.14.0-390.el9.x86_64.rpm 25 MB/s | 20 MB 00:00 2023-12-11T19:54:35.763 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 11 MB/s | 38 MB 00:03 2023-12-11T19:54:36.717 INFO:teuthology.orchestra.run.smithi133.stdout:(3/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 14 MB/s | 38 MB 00:02 2023-12-11T19:54:37.042 INFO:teuthology.orchestra.run.smithi133.stdout:(4/4): kernel-modules-core-5.14.0-390.el9.x86_6 12 MB/s | 32 MB 00:02 2023-12-11T19:54:37.042 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T19:54:37.043 INFO:teuthology.orchestra.run.smithi133.stdout:Total 30 MB/s | 96 MB 00:03 2023-12-11T19:54:37.896 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T19:54:38.268 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T19:54:38.268 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T19:54:39.313 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-core-5.14.0-390.el9.x86_6 6.4 MB/s | 32 MB 00:05 2023-12-11T19:54:39.571 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-core-5.14.0-390.el9.x86_64.rpm 2.8 MB/s | 20 MB 00:07 2023-12-11T19:54:39.572 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T19:54:39.572 INFO:teuthology.orchestra.run.smithi086.stdout:Total 13 MB/s | 96 MB 00:07 2023-12-11T19:54:39.572 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T19:54:39.572 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T19:54:40.427 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T19:54:40.761 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T19:54:40.761 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T19:54:42.068 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T19:54:42.069 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T19:54:42.214 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T19:54:42.847 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-11T19:54:42.958 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-11T19:54:44.981 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T19:54:45.644 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-11T19:54:45.697 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-11T19:54:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-11T19:54:45.878 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-11T19:54:48.580 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-11T19:54:48.770 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-11T19:54:50.486 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-11T19:54:50.617 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:54:53.399 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-11T19:54:53.530 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:54:55.232 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:54:58.169 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:23.854 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:23.896 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:24.341 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:24.341 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-11T19:55:24.341 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-11T19:55:24.341 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-11T19:55:24.784 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:24.784 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T19:55:24.784 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T19:55:24.784 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-11T19:55:24.784 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-11T19:55:24.784 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-11T19:55:24.784 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-11T19:55:24.785 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T19:55:24.785 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T19:55:25.258 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2023-12-11T19:55:25.403 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-11T19:55:25.403 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-11T19:55:25.403 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-11T19:55:25.404 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-11T19:55:25.404 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-11T19:55:25.404 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-11T19:55:25.404 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2023-12-11T19:55:25.404 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2023-12-11T19:55:25.522 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-11T19:55:25.523 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-11T19:55:25.523 DEBUG:teuthology.orchestra.run.smithi133:> sudo rpm -qi grub2-tools 2023-12-11T19:55:25.568 INFO:teuthology.orchestra.run.smithi133.stdout:Name : grub2-tools 2023-12-11T19:55:25.568 INFO:teuthology.orchestra.run.smithi133.stdout:Epoch : 1 2023-12-11T19:55:25.568 INFO:teuthology.orchestra.run.smithi133.stdout:Version : 2.06 2023-12-11T19:55:25.568 INFO:teuthology.orchestra.run.smithi133.stdout:Release : 68.el9 2023-12-11T19:55:25.569 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: x86_64 2023-12-11T19:55:25.569 INFO:teuthology.orchestra.run.smithi133.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-11T19:55:25.569 INFO:teuthology.orchestra.run.smithi133.stdout:Group : Unspecified 2023-12-11T19:55:25.569 INFO:teuthology.orchestra.run.smithi133.stdout:Size : 8264254 2023-12-11T19:55:25.569 INFO:teuthology.orchestra.run.smithi133.stdout:License : GPLv3+ 2023-12-11T19:55:25.569 INFO:teuthology.orchestra.run.smithi133.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-11T19:55:25.570 INFO:teuthology.orchestra.run.smithi133.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-11T19:55:25.570 INFO:teuthology.orchestra.run.smithi133.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-11T19:55:25.570 INFO:teuthology.orchestra.run.smithi133.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-11T19:55:25.570 INFO:teuthology.orchestra.run.smithi133.stdout:Packager : builder@centos.org 2023-12-11T19:55:25.570 INFO:teuthology.orchestra.run.smithi133.stdout:Vendor : CentOS 2023-12-11T19:55:25.570 INFO:teuthology.orchestra.run.smithi133.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-11T19:55:25.571 INFO:teuthology.orchestra.run.smithi133.stdout:Summary : Support tools for GRUB. 2023-12-11T19:55:25.571 INFO:teuthology.orchestra.run.smithi133.stdout:Description : 2023-12-11T19:55:25.571 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T19:55:25.571 INFO:teuthology.orchestra.run.smithi133.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-11T19:55:25.571 INFO:teuthology.orchestra.run.smithi133.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-11T19:55:25.571 INFO:teuthology.orchestra.run.smithi133.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-11T19:55:25.571 INFO:teuthology.orchestra.run.smithi133.stdout:hardware devices. 2023-12-11T19:55:25.572 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T19:55:25.572 INFO:teuthology.orchestra.run.smithi133.stdout:This subpackage provides tools for support of all platforms. 2023-12-11T19:55:25.573 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-11T19:55:25.574 INFO:teuthology.task.kernel:Updating grub on smithi133 to boot 5.14.0-390.el9.x86_64 2023-12-11T19:55:25.574 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-11T19:55:26.121 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2023-12-11T19:55:27.210 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:27.252 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:27.543 INFO:teuthology.orchestra.run.smithi133.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-11T19:55:27.583 INFO:teuthology.orchestra.run.smithi133.stderr:done 2023-12-11T19:55:27.585 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2023-12-11T19:55:27.599 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.WbTV0qnAoN 2023-12-11T19:55:27.600 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WbTV0qnAoN 2023-12-11T19:55:27.642 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:27.645 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-11T19:55:27.645 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-11T19:55:27.645 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-11T19:55:27.669 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.WbTV0qnAoN 2023-12-11T19:55:27.830 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.WbTV0qnAoN is 6KB 2023-12-11T19:55:27.881 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.WbTV0qnAoN 2023-12-11T19:55:27.896 DEBUG:teuthology.orchestra.run.smithi133:> sudo /bin/ls /boot/loader/entries || true 2023-12-11T19:55:27.965 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-11T19:55:27.965 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-11T19:55:27.965 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-11T19:55:27.965 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-11T19:55:27.967 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T19:55:28.053 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T19:55:28.125 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2023-12-11T19:55:28.564 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-12-11T19:55:28.716 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-11T19:55:28.718 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-11T19:55:28.718 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-11T19:55:28.718 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-11T19:55:28.718 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-11T19:55:28.719 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-11T19:55:28.719 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2023-12-11T19:55:28.719 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-12-11T19:55:28.878 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-11T19:55:28.879 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-11T19:55:28.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2023-12-11T19:55:28.923 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2023-12-11T19:55:28.924 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T19:55:28.925 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2023-12-11T19:55:28.926 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-11T19:55:28.926 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-390.el9.x86_64 2023-12-11T19:55:28.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-11T19:55:29.531 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2023-12-11T19:55:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-11T19:55:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:done 2023-12-11T19:55:30.956 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-12-11T19:55:30.972 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.VjPyvHUfgr 2023-12-11T19:55:30.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VjPyvHUfgr 2023-12-11T19:55:31.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.VjPyvHUfgr 2023-12-11T19:55:31.180 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.VjPyvHUfgr is 6KB 2023-12-11T19:55:31.228 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.VjPyvHUfgr 2023-12-11T19:55:31.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2023-12-11T19:55:31.314 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-11T19:55:31.314 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-11T19:55:31.314 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-11T19:55:31.314 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-11T19:55:31.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-11T19:55:31.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2023-12-11T19:55:58.154 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-11T19:55:58.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-12-11T19:55:58.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T19:56:01.479 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-11T19:56:01.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-12-11T19:56:01.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T19:56:16.610 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-12-11T19:56:25.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-12-11T19:56:25.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T19:56:28.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-12-11T19:56:36.066 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-12-11T19:56:40.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-12-11T19:56:40.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T19:56:45.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-12-11T19:56:45.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T19:57:03.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-12-11T19:57:12.582 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-12-11T19:57:13.071 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-12-11T19:57:13.071 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-390.el9.x86_64"... 2023-12-11T19:57:13.072 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-12-11T19:57:13.098 INFO:teuthology.orchestra.run.smithi133.stdout:5.14.0-390.el9.x86_64 2023-12-11T19:57:13.098 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-11T19:57:13.098 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-11T19:57:13.098 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-11T19:57:14.099 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-11T19:57:14.100 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-11T19:57:14.237 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2023-12-11T19:57:14.322 DEBUG:teuthology.parallel:result is None 2023-12-11T19:57:15.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-12-11T19:57:15.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T19:57:15.949 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-12-11T19:57:16.275 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-12-11T19:57:16.275 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-390.el9.x86_64"... 2023-12-11T19:57:16.276 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-12-11T19:57:16.290 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-390.el9.x86_64 2023-12-11T19:57:16.290 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-11T19:57:16.290 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-11T19:57:16.290 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-11T19:57:17.291 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-11T19:57:17.291 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-11T19:57:17.352 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2023-12-11T19:57:17.406 DEBUG:teuthology.parallel:result is None 2023-12-11T19:57:17.407 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-11T19:57:17.895 INFO:teuthology.task.internal:Creating test directory... 2023-12-11T19:57:17.895 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-11T19:57:17.898 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-11T19:57:17.917 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-11T19:57:17.975 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-11T19:57:17.986 INFO:teuthology.task.internal:Creating archive directory... 2023-12-11T19:57:17.987 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-11T19:57:17.991 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-11T19:57:18.113 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-11T19:57:18.120 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-11T19:57:18.120 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-11T19:57:18.122 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-11T19:57:18.158 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-11T19:57:18.172 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-11T19:57:18.179 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-11T19:57:18.192 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-11T19:57:18.193 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-11T19:57:18.201 INFO:teuthology.task.internal:Configuring sudo... 2023-12-11T19:57:18.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-11T19:57:18.203 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-11T19:57:18.244 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-11T19:57:18.252 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-11T19:57:18.253 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-11T19:57:18.273 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-11T19:57:18.302 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-11T19:57:18.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-11T19:57:18.421 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-11T19:57:18.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-11T19:57:18.544 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T19:57:18.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-11T19:57:18.610 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-11T19:57:18.654 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-11T19:57:18.715 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-11T19:57:18.760 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-11T19:57:18.802 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T19:57:18.803 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-11T19:57:18.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2023-12-11T19:57:18.873 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2023-12-11T19:57:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-11T19:57:18.954 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-11T19:57:19.170 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-11T19:57:19.179 INFO:teuthology.task.internal:Starting timer... 2023-12-11T19:57:19.179 INFO:teuthology.run_tasks:Running task pcp... 2023-12-11T19:57:19.189 INFO:teuthology.run_tasks:Running task selinux... 2023-12-11T19:57:19.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2023-12-11T19:57:19.256 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2023-12-11T19:57:19.271 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2023-12-11T19:57:19.355 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: 2023-12-11T19:57:19.357 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-11T19:57:19.357 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2023-12-11T19:57:19.378 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2023-12-11T19:57:19.379 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2023-12-11T19:57:19.397 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2023-12-11T19:57:19.397 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi133.front.sepia.ceph.com': 'permissive'} 2023-12-11T19:57:19.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-11T19:57:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T19:57:19.446 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-11T19:57:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T19:57:19.474 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-11T19:57:19.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2023-12-11T19:57:19.522 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2023-12-11T19:57:19.557 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-11T19:57:19.567 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-11T19:57:19.567 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-11T19:57:19.585 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-11T19:57:19.586 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi133.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-11T20:02:56.304 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi133.front.sepia.ceph.com')] 2023-12-11T20:02:56.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-12-11T20:02:56.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T20:02:56.389 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-12-11T20:02:56.457 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-12-11T20:02:56.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-12-11T20:02:56.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-11T20:02:56.541 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-12-11T20:02:56.614 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-12-11T20:02:56.614 INFO:teuthology.run_tasks:Running task clock... 2023-12-11T20:02:56.628 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-11T20:02:56.629 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-11T20:02:56.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-11T20:02:56.631 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-11T20:02:56.631 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-11T20:02:56.660 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-11T20:02:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-11T20:02:56.687 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-11T20:02:56.703 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-11T20:02:56.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2023-12-11T20:02:56.720 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2023-12-11T20:02:56.732 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: ntpd: command not found 2023-12-11T20:02:56.736 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-11T20:02:56.746 INFO:teuthology.orchestra.run.smithi133.stdout:506 Cannot talk to daemon 2023-12-11T20:02:56.753 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-11T20:02:56.763 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-11T20:02:56.779 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-11T20:02:56.795 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2023-12-11T20:02:56.798 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-11T20:02:56.799 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-12-11T20:02:56.799 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:02:56.799 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:02:56.799 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:02:56.799 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:02:56.822 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2023-12-11T20:02:56.824 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-11T20:02:56.825 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2023-12-11T20:02:56.825 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:02:56.825 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:02:56.825 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:02:56.825 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:02:56.826 INFO:teuthology.run_tasks:Running task install... 2023-12-11T20:02:56.837 DEBUG:teuthology.task.install:project ceph 2023-12-11T20:02:56.838 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'}} 2023-12-11T20:02:56.838 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'} 2023-12-11T20:02:56.838 INFO:teuthology.task.install:Using flavor: default 2023-12-11T20:02:56.843 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-12-11T20:02:56.844 INFO:teuthology.task.install:extra packages: [] 2023-12-11T20:02:56.844 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'tag': None, 'wait_for_package': False} 2023-12-11T20:02:56.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-11T20:02:56.846 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'tag': None, 'wait_for_package': False} 2023-12-11T20:02:56.846 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-11T20:02:56.993 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/ 2023-12-11T20:02:56.993 INFO:teuthology.task.install.rpm:Package version is 19.0.0-83.gc56ad3ea 2023-12-11T20:02:56.996 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/ 2023-12-11T20:02:56.997 INFO:teuthology.task.install.rpm:Package version is 19.0.0-83.gc56ad3ea 2023-12-11T20:02:57.197 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-11T20:02:57.198 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:02:57.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-11T20:02:57.226 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-11T20:02:57.227 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-11T20:02:57.261 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-11T20:02:57.261 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:02:57.261 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-11T20:02:57.289 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-11T20:02:57.289 DEBUG:teuthology.orchestra.run.smithi133:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-11T20:02:57.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-11T20:02:57.357 DEBUG:teuthology.orchestra.run.smithi133:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-11T20:02:57.374 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-11T20:02:57.437 DEBUG:teuthology.orchestra.run.smithi133:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-11T20:02:57.442 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2023-12-11T20:02:57.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2023-12-11T20:02:57.503 INFO:teuthology.orchestra.run.smithi133.stdout:check_obsoletes = 1 2023-12-11T20:02:57.505 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum clean all 2023-12-11T20:02:57.680 INFO:teuthology.orchestra.run.smithi086.stdout:73 files removed 2023-12-11T20:02:57.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2023-12-11T20:02:57.749 INFO:teuthology.orchestra.run.smithi133.stdout:73 files removed 2023-12-11T20:02:57.776 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw 2023-12-11T20:02:58.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 117 kB/s | 79 kB 00:00 2023-12-11T20:02:58.850 INFO:teuthology.orchestra.run.smithi133.stdout:ceph packages for x86_64 105 kB/s | 79 kB 00:00 2023-12-11T20:02:59.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 13 kB/s | 9.5 kB 00:00 2023-12-11T20:02:59.634 INFO:teuthology.orchestra.run.smithi133.stdout:ceph noarch packages 13 kB/s | 9.5 kB 00:00 2023-12-11T20:03:00.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2023-12-11T20:03:00.334 INFO:teuthology.orchestra.run.smithi133.stdout:ceph source packages 2.8 kB/s | 1.9 kB 00:00 2023-12-11T20:03:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-11T20:03:00.711 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - AppStream 19 MB/s | 6.4 MB 00:00 2023-12-11T20:03:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2023-12-11T20:03:02.112 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2023-12-11T20:03:04.186 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 7.9 MB 00:01 2023-12-11T20:03:05.917 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 7.9 MB 00:03 2023-12-11T20:03:07.695 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 18 MB 00:02 2023-12-11T20:03:13.320 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 5.8 MB 00:02 2023-12-11T20:03:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 18 MB 00:06 2023-12-11T20:03:15.137 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2023-12-11T20:03:16.433 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2023-12-11T20:03:17.493 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 5.8 MB 00:00 2023-12-11T20:03:20.177 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2023-12-11T20:03:21.006 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 29 MB/s | 20 MB 00:00 2023-12-11T20:03:21.247 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-12-11T20:03:23.825 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 8.6 M 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrading: 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.4 M 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.1 M 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.2 M 2023-12-11T20:03:23.832 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 19 M 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 24 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 743 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 499 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 4.8 M 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-11T20:03:23.833 INFO:teuthology.orchestra.run.smithi133.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 45 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 129 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 162 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 321 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 298 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 99 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout:Install 27 Packages 2023-12-11T20:03:23.834 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrade 2 Packages 2023-12-11T20:03:23.835 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:23.835 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 55 M 2023-12-11T20:03:23.835 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:03:25.614 INFO:teuthology.orchestra.run.smithi133.stdout:(1/29): ceph-base-19.0.0-83.gc56ad3ea.el9.x86_6 3.1 MB/s | 5.2 MB 00:01 2023-12-11T20:03:25.706 INFO:teuthology.orchestra.run.smithi133.stdout:(2/29): ceph-selinux-19.0.0-83.gc56ad3ea.el9.x8 263 kB/s | 24 kB 00:00 2023-12-11T20:03:25.812 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-12-11T20:03:25.898 INFO:teuthology.orchestra.run.smithi133.stdout:(3/29): libcephfs2-19.0.0-83.gc56ad3ea.el9.x86_ 3.8 MB/s | 743 kB 00:00 2023-12-11T20:03:26.007 INFO:teuthology.orchestra.run.smithi133.stdout:(4/29): libradosstriper1-19.0.0-83.gc56ad3ea.el 4.5 MB/s | 499 kB 00:00 2023-12-11T20:03:28.391 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:03:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:03:28.437 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:03:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:03:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 8.6 M 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.4 M 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.1 M 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.2 M 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 19 M 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 24 k 2023-12-11T20:03:28.438 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 743 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 499 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 4.8 M 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-11T20:03:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 45 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 129 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 162 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 321 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 298 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 99 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-11T20:03:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-11T20:03:28.441 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:28.441 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:03:28.441 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:03:28.441 INFO:teuthology.orchestra.run.smithi086.stdout:Install 27 Packages 2023-12-11T20:03:28.441 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2023-12-11T20:03:28.441 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:28.442 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 55 M 2023-12-11T20:03:28.442 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:03:31.029 INFO:teuthology.orchestra.run.smithi086.stdout:(1/29): ceph-base-19.0.0-83.gc56ad3ea.el9.x86_6 2.3 MB/s | 5.2 MB 00:02 2023-12-11T20:03:31.088 INFO:teuthology.orchestra.run.smithi086.stdout:(2/29): ceph-selinux-19.0.0-83.gc56ad3ea.el9.x8 413 kB/s | 24 kB 00:00 2023-12-11T20:03:31.346 INFO:teuthology.orchestra.run.smithi086.stdout:(3/29): ceph-radosgw-19.0.0-83.gc56ad3ea.el9.x8 3.3 MB/s | 8.6 MB 00:02 2023-12-11T20:03:31.380 INFO:teuthology.orchestra.run.smithi086.stdout:(4/29): libcephfs2-19.0.0-83.gc56ad3ea.el9.x86_ 2.5 MB/s | 743 kB 00:00 2023-12-11T20:03:31.489 INFO:teuthology.orchestra.run.smithi086.stdout:(5/29): libradosstriper1-19.0.0-83.gc56ad3ea.el 3.4 MB/s | 499 kB 00:00 2023-12-11T20:03:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:(6/29): python3-ceph-argparse-19.0.0-83.gc56ad3 542 kB/s | 45 kB 00:00 2023-12-11T20:03:31.666 INFO:teuthology.orchestra.run.smithi086.stdout:(7/29): python3-ceph-common-19.0.0-83.gc56ad3ea 1.4 MB/s | 129 kB 00:00 2023-12-11T20:03:31.756 INFO:teuthology.orchestra.run.smithi086.stdout:(8/29): python3-cephfs-19.0.0-83.gc56ad3ea.el9. 1.7 MB/s | 162 kB 00:00 2023-12-11T20:03:31.856 INFO:teuthology.orchestra.run.smithi086.stdout:(9/29): python3-rados-19.0.0-83.gc56ad3ea.el9.x 3.1 MB/s | 321 kB 00:00 2023-12-11T20:03:31.990 INFO:teuthology.orchestra.run.smithi086.stdout:(10/29): librgw2-19.0.0-83.gc56ad3ea.el9.x86_64 7.8 MB/s | 4.8 MB 00:00 2023-12-11T20:03:32.015 INFO:teuthology.orchestra.run.smithi086.stdout:(11/29): python3-rbd-19.0.0-83.gc56ad3ea.el9.x8 1.8 MB/s | 298 kB 00:00 2023-12-11T20:03:32.074 INFO:teuthology.orchestra.run.smithi086.stdout:(12/29): python3-rgw-19.0.0-83.gc56ad3ea.el9.x8 1.2 MB/s | 99 kB 00:00 2023-12-11T20:03:32.125 INFO:teuthology.orchestra.run.smithi086.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 971 kB/s | 106 kB 00:00 2023-12-11T20:03:32.149 INFO:teuthology.orchestra.run.smithi086.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2023-12-11T20:03:32.175 INFO:teuthology.orchestra.run.smithi086.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2023-12-11T20:03:32.225 INFO:teuthology.orchestra.run.smithi086.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-12-11T20:03:32.250 INFO:teuthology.orchestra.run.smithi086.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-12-11T20:03:32.267 INFO:teuthology.orchestra.run.smithi086.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 992 kB/s | 41 kB 00:00 2023-12-11T20:03:32.917 INFO:teuthology.orchestra.run.smithi086.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 463 kB/s | 309 kB 00:00 2023-12-11T20:03:33.043 INFO:teuthology.orchestra.run.smithi086.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2023-12-11T20:03:33.160 INFO:teuthology.orchestra.run.smithi086.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 419 kB/s | 49 kB 00:00 2023-12-11T20:03:33.268 INFO:teuthology.orchestra.run.smithi086.stdout:(22/29): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2023-12-11T20:03:33.285 INFO:teuthology.orchestra.run.smithi086.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 544 kB/s | 67 kB 00:00 2023-12-11T20:03:33.519 INFO:teuthology.orchestra.run.smithi086.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2023-12-11T20:03:33.548 INFO:teuthology.orchestra.run.smithi086.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 730 kB/s | 192 kB 00:00 2023-12-11T20:03:33.732 INFO:teuthology.orchestra.run.smithi086.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 7.5 MB/s | 1.6 MB 00:00 2023-12-11T20:03:34.057 INFO:teuthology.orchestra.run.smithi086.stdout:(27/29): ceph-common-19.0.0-83.gc56ad3ea.el9.x8 3.6 MB/s | 19 MB 00:05 2023-12-11T20:03:34.173 INFO:teuthology.orchestra.run.smithi133.stdout:(5/29): librgw2-19.0.0-83.gc56ad3ea.el9.x86_64. 599 kB/s | 4.8 MB 00:08 2023-12-11T20:03:34.524 INFO:teuthology.orchestra.run.smithi086.stdout:(28/29): librbd1-19.0.0-83.gc56ad3ea.el9.x86_64 4.0 MB/s | 3.1 MB 00:00 2023-12-11T20:03:34.591 INFO:teuthology.orchestra.run.smithi086.stdout:(29/29): librados2-19.0.0-83.gc56ad3ea.el9.x86_ 3.2 MB/s | 3.4 MB 00:01 2023-12-11T20:03:34.597 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:03:34.597 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.0 MB/s | 55 MB 00:06 2023-12-11T20:03:34.597 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:03:34.633 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:03:34.633 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:03:34.648 INFO:teuthology.orchestra.run.smithi133.stdout:(6/29): ceph-radosgw-19.0.0-83.gc56ad3ea.el9.x8 822 kB/s | 8.6 MB 00:10 2023-12-11T20:03:34.665 INFO:teuthology.orchestra.run.smithi133.stdout:(7/29): python3-ceph-argparse-19.0.0-83.gc56ad3 92 kB/s | 45 kB 00:00 2023-12-11T20:03:34.715 INFO:teuthology.orchestra.run.smithi133.stdout:(8/29): python3-ceph-common-19.0.0-83.gc56ad3ea 1.9 MB/s | 129 kB 00:00 2023-12-11T20:03:34.742 INFO:teuthology.orchestra.run.smithi133.stdout:(9/29): python3-cephfs-19.0.0-83.gc56ad3ea.el9. 2.1 MB/s | 162 kB 00:00 2023-12-11T20:03:34.794 INFO:teuthology.orchestra.run.smithi133.stdout:(10/29): python3-rados-19.0.0-83.gc56ad3ea.el9. 4.0 MB/s | 321 kB 00:00 2023-12-11T20:03:34.853 INFO:teuthology.orchestra.run.smithi133.stdout:(11/29): python3-rbd-19.0.0-83.gc56ad3ea.el9.x8 2.6 MB/s | 298 kB 00:00 2023-12-11T20:03:34.878 INFO:teuthology.orchestra.run.smithi133.stdout:(12/29): python3-rgw-19.0.0-83.gc56ad3ea.el9.x8 1.2 MB/s | 99 kB 00:00 2023-12-11T20:03:34.987 INFO:teuthology.orchestra.run.smithi133.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 792 kB/s | 106 kB 00:00 2023-12-11T20:03:35.012 INFO:teuthology.orchestra.run.smithi133.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2023-12-11T20:03:35.037 INFO:teuthology.orchestra.run.smithi133.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2023-12-11T20:03:35.087 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:03:35.087 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:03:35.088 INFO:teuthology.orchestra.run.smithi133.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-12-11T20:03:35.113 INFO:teuthology.orchestra.run.smithi133.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-12-11T20:03:35.138 INFO:teuthology.orchestra.run.smithi133.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2023-12-11T20:03:35.264 INFO:teuthology.orchestra.run.smithi133.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2023-12-11T20:03:35.305 INFO:teuthology.orchestra.run.smithi133.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2023-12-11T20:03:35.347 INFO:teuthology.orchestra.run.smithi133.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2023-12-11T20:03:35.389 INFO:teuthology.orchestra.run.smithi133.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2023-12-11T20:03:35.481 INFO:teuthology.orchestra.run.smithi133.stdout:(23/29): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2023-12-11T20:03:35.523 INFO:teuthology.orchestra.run.smithi133.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2023-12-11T20:03:35.549 INFO:teuthology.orchestra.run.smithi133.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2023-12-11T20:03:35.624 INFO:teuthology.orchestra.run.smithi133.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-12-11T20:03:36.208 INFO:teuthology.orchestra.run.smithi133.stdout:(27/29): librbd1-19.0.0-83.gc56ad3ea.el9.x86_64 5.4 MB/s | 3.1 MB 00:00 2023-12-11T20:03:36.391 INFO:teuthology.orchestra.run.smithi133.stdout:(28/29): librados2-19.0.0-83.gc56ad3ea.el9.x86_ 4.0 MB/s | 3.4 MB 00:00 2023-12-11T20:03:36.444 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:03:36.518 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-11T20:03:36.545 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-11T20:03:36.604 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-11T20:03:36.923 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-11T20:03:36.974 INFO:teuthology.orchestra.run.smithi133.stdout:(29/29): ceph-common-19.0.0-83.gc56ad3ea.el9.x8 1.5 MB/s | 19 MB 00:13 2023-12-11T20:03:36.979 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:03:36.979 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.2 MB/s | 55 MB 00:13 2023-12-11T20:03:36.980 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:03:36.987 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-11T20:03:37.016 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:03:37.016 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:03:37.077 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-11T20:03:37.166 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/31 2023-12-11T20:03:37.198 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-11T20:03:37.266 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-11T20:03:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-11T20:03:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x8 9/31 2023-12-11T20:03:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 10/31 2023-12-11T20:03:37.432 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-11T20:03:37.466 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:03:37.467 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:03:37.480 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-11T20:03:37.587 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-11T20:03:37.623 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-11T20:03:37.688 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-11T20:03:37.720 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-11T20:03:37.749 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-11T20:03:37.782 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-11T20:03:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-11T20:03:38.120 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 19/31 2023-12-11T20:03:38.210 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 19/31 2023-12-11T20:03:38.251 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 20/31 2023-12-11T20:03:38.287 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-11T20:03:38.721 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-11T20:03:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 23/31 2023-12-11T20:03:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 23/31 2023-12-11T20:03:38.870 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:03:38.895 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 24/31 2023-12-11T20:03:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_ 25/31 2023-12-11T20:03:38.970 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-11T20:03:39.012 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-11T20:03:39.063 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-11T20:03:39.434 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-11T20:03:39.479 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-11T20:03:39.570 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-11T20:03:39.665 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/31 2023-12-11T20:03:39.696 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-11T20:03:39.759 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-11T20:03:39.871 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-11T20:03:39.932 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x8 9/31 2023-12-11T20:03:40.000 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 10/31 2023-12-11T20:03:40.024 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-11T20:03:40.089 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-11T20:03:40.197 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-11T20:03:40.240 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-11T20:03:40.289 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-11T20:03:40.321 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-11T20:03:40.350 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-11T20:03:40.386 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-11T20:03:40.691 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-11T20:03:40.721 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 19/31 2023-12-11T20:03:40.811 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 19/31 2023-12-11T20:03:40.842 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 20/31 2023-12-11T20:03:40.878 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-11T20:03:41.129 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-11T20:03:41.173 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-11T20:03:41.300 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-11T20:03:41.332 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 23/31 2023-12-11T20:03:41.386 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 23/31 2023-12-11T20:03:41.453 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 24/31 2023-12-11T20:03:41.484 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_ 25/31 2023-12-11T20:03:41.736 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-11T20:03:41.778 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 27/31 2023-12-11T20:03:41.864 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 27/31 2023-12-11T20:03:41.865 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-11T20:03:41.865 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-11T20:03:41.865 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:41.902 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-11T20:03:43.570 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-11T20:03:43.606 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-11T20:03:44.157 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-11T20:03:44.192 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 27/31 2023-12-11T20:03:44.278 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 27/31 2023-12-11T20:03:44.278 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-11T20:03:44.278 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-11T20:03:44.278 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:44.319 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-11T20:03:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-11T20:03:51.332 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2023-12-11T20:03:51.332 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2023-12-11T20:03:51.332 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2023-12-11T20:03:51.332 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2023-12-11T20:03:51.333 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2023-12-11T20:03:51.333 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2023-12-11T20:03:51.333 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2023-12-11T20:03:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 29/31 2023-12-11T20:03:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 29/31 2023-12-11T20:03:52.175 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:03:52.175 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-11T20:03:52.175 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-11T20:03:52.176 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-11T20:03:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-11T20:03:52.262 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-11T20:03:52.262 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-11T20:03:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 3/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 4/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x8 8/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_ 9/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 10/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 12/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 13/31 2023-12-11T20:03:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-11T20:03:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-11T20:03:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-11T20:03:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 30/31 2023-12-11T20:03:53.444 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-11T20:03:53.444 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /sys 2023-12-11T20:03:53.444 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /proc 2023-12-11T20:03:53.444 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /mnt 2023-12-11T20:03:53.444 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /var/tmp 2023-12-11T20:03:53.444 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /home 2023-12-11T20:03:53.445 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /root 2023-12-11T20:03:53.445 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /tmp 2023-12-11T20:03:53.445 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:54.267 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 29/31 2023-12-11T20:03:54.291 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 29/31 2023-12-11T20:03:54.291 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:03:54.291 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-11T20:03:54.291 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-11T20:03:54.291 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-11T20:03:54.292 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:54.293 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-11T20:03:54.378 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-11T20:03:54.378 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-11T20:03:54.522 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-11T20:03:54.522 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:54.522 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-11T20:03:54.523 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-11T20:03:54.524 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.525 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.525 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:54.525 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-3.el9.x86_64 2023-12-11T20:03:54.525 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-11T20:03:54.525 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:54.525 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:03:54.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2023-12-11T20:03:55.430 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:30 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:03:55.475 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-11T20:03:55.475 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/31 2023-12-11T20:03:55.475 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/31 2023-12-11T20:03:55.475 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 3/31 2023-12-11T20:03:55.475 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 4/31 2023-12-11T20:03:55.475 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-11T20:03:55.475 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x8 8/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_ 9/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 10/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 12/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 13/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-11T20:03:55.476 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-11T20:03:55.477 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-11T20:03:55.477 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-11T20:03:55.477 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-11T20:03:55.477 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-11T20:03:55.477 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-11T20:03:55.477 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-11T20:03:55.477 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-11T20:03:55.478 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-11T20:03:55.478 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 30/31 2023-12-11T20:03:55.574 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:03:55.575 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:03:55.575 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:03:55.575 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:03:55.575 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:03:55.575 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 45 M 2023-12-11T20:03:55.575 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:03:55.575 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-11T20:03:55.575 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 164 k 2023-12-11T20:03:55.576 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-11T20:03:55.576 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-11T20:03:55.576 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-11T20:03:55.576 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:55.576 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:03:55.576 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:03:55.576 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-12-11T20:03:55.576 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:03:55.577 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 46 M 2023-12-11T20:03:55.577 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 199 M 2023-12-11T20:03:55.577 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:03:55.726 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-11T20:03:55.809 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2023-12-11T20:03:55.884 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2023-12-11T20:03:55.926 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-11T20:03:56.018 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): libcephsqlite-19.0.0-83.gc56ad3ea.el9.x8 372 kB/s | 164 kB 00:00 2023-12-11T20:03:56.472 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout:Upgraded: 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-11T20:03:56.473 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-11T20:03:56.474 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.475 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.475 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:03:56.475 INFO:teuthology.orchestra.run.smithi133.stdout: re2-1:20211101-3.el9.x86_64 2023-12-11T20:03:56.475 INFO:teuthology.orchestra.run.smithi133.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-11T20:03:56.475 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:56.475 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:03:56.729 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-test 2023-12-11T20:03:57.298 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:36 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:03:57.444 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:03:57.445 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:03:57.445 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:03:57.445 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:03:57.445 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:03:57.445 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 45 M 2023-12-11T20:03:57.445 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:03:57.445 INFO:teuthology.orchestra.run.smithi133.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 164 k 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2023-12-11T20:03:57.446 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:03:57.447 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 46 M 2023-12-11T20:03:57.447 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 199 M 2023-12-11T20:03:57.447 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:03:57.576 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 186 kB 00:00 2023-12-11T20:03:57.651 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.9 MB/s | 217 kB 00:00 2023-12-11T20:03:57.718 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2023-12-11T20:03:57.761 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-11T20:03:57.910 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): libcephsqlite-19.0.0-83.gc56ad3ea.el9.x8 354 kB/s | 164 kB 00:00 2023-12-11T20:04:00.185 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-19.0.0-83.gc56ad3ea.el9.x86_64 9.8 MB/s | 45 MB 00:04 2023-12-11T20:04:00.188 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:00.189 INFO:teuthology.orchestra.run.smithi086.stdout:Total 10 MB/s | 46 MB 00:04 2023-12-11T20:04:00.189 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:00.208 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:00.209 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:01.150 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-11T20:04:01.268 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-11T20:04:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-11T20:04:01.343 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-11T20:04:01.379 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/6 2023-12-11T20:04:05.118 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): ceph-test-19.0.0-83.gc56ad3ea.el9.x86_64 5.9 MB/s | 45 MB 00:07 2023-12-11T20:04:05.121 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:05.122 INFO:teuthology.orchestra.run.smithi133.stdout:Total 6.0 MB/s | 46 MB 00:07 2023-12-11T20:04:05.122 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:04:05.142 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:04:05.142 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:04:05.537 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:04:05.537 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:04:05.761 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/6 2023-12-11T20:04:05.845 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/6 2023-12-11T20:04:06.100 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:04:06.174 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-11T20:04:06.211 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-11T20:04:06.242 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-11T20:04:06.285 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-11T20:04:06.313 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/6 2023-12-11T20:04:08.188 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/6 2023-12-11T20:04:08.188 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/6 2023-12-11T20:04:08.188 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/6 2023-12-11T20:04:08.188 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-11T20:04:08.188 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-11T20:04:08.188 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-11T20:04:08.584 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-11T20:04:08.585 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:08.585 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:08.585 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 jq-1.6-16.el9.x86_64 2023-12-11T20:04:08.585 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2023-12-11T20:04:08.585 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2023-12-11T20:04:08.585 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:08.585 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:08.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2023-12-11T20:04:09.446 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:44 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:09.612 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:09.614 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2023-12-11T20:04:09.614 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:04:09.614 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 6.5 k 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 2.1 M 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.5 M 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 243 k 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 4.4 M 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 18 M 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-11T20:04:09.615 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-11T20:04:09.616 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout:Install 36 Packages 2023-12-11T20:04:09.617 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:09.618 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 30 M 2023-12-11T20:04:09.618 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 112 M 2023-12-11T20:04:09.618 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:09.978 INFO:teuthology.orchestra.run.smithi086.stdout:(1/36): ceph-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 27 kB/s | 6.5 kB 00:00 2023-12-11T20:04:10.170 INFO:teuthology.orchestra.run.smithi086.stdout:(2/36): ceph-mgr-19.0.0-83.gc56ad3ea.el9.x86_64 3.4 MB/s | 1.5 MB 00:00 2023-12-11T20:04:10.220 INFO:teuthology.orchestra.run.smithi086.stdout:(3/36): ceph-mds-19.0.0-83.gc56ad3ea.el9.x86_64 4.3 MB/s | 2.1 MB 00:00 2023-12-11T20:04:10.320 INFO:teuthology.orchestra.run.smithi086.stdout:(4/36): ceph-mgr-modules-core-19.0.0-83.gc56ad3 2.4 MB/s | 243 kB 00:00 2023-12-11T20:04:10.421 INFO:teuthology.orchestra.run.smithi086.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-12-11T20:04:10.454 INFO:teuthology.orchestra.run.smithi086.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2023-12-11T20:04:10.480 INFO:teuthology.orchestra.run.smithi086.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-12-11T20:04:10.505 INFO:teuthology.orchestra.run.smithi086.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2023-12-11T20:04:10.539 INFO:teuthology.orchestra.run.smithi086.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-11T20:04:10.580 INFO:teuthology.orchestra.run.smithi086.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-11T20:04:10.638 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/6 2023-12-11T20:04:10.647 INFO:teuthology.orchestra.run.smithi086.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-11T20:04:10.681 INFO:teuthology.orchestra.run.smithi086.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-11T20:04:10.706 INFO:teuthology.orchestra.run.smithi086.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 4.9 MB/s | 124 kB 00:00 2023-12-11T20:04:10.720 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/6 2023-12-11T20:04:10.740 INFO:teuthology.orchestra.run.smithi086.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-11T20:04:10.773 INFO:teuthology.orchestra.run.smithi086.stdout:(15/36): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-11T20:04:10.874 INFO:teuthology.orchestra.run.smithi086.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2023-12-11T20:04:11.074 INFO:teuthology.orchestra.run.smithi086.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 756 kB/s | 151 kB 00:00 2023-12-11T20:04:11.133 INFO:teuthology.orchestra.run.smithi086.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 743 kB/s | 43 kB 00:00 2023-12-11T20:04:11.216 INFO:teuthology.orchestra.run.smithi086.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2023-12-11T20:04:11.317 INFO:teuthology.orchestra.run.smithi086.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2023-12-11T20:04:11.367 INFO:teuthology.orchestra.run.smithi086.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2023-12-11T20:04:11.425 INFO:teuthology.orchestra.run.smithi086.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2023-12-11T20:04:11.475 INFO:teuthology.orchestra.run.smithi086.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2023-12-11T20:04:11.526 INFO:teuthology.orchestra.run.smithi086.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2023-12-11T20:04:11.576 INFO:teuthology.orchestra.run.smithi086.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 394 kB/s | 20 kB 00:00 2023-12-11T20:04:11.751 INFO:teuthology.orchestra.run.smithi086.stdout:(26/36): ceph-osd-19.0.0-83.gc56ad3ea.el9.x86_6 11 MB/s | 18 MB 00:01 2023-12-11T20:04:11.776 INFO:teuthology.orchestra.run.smithi086.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 231 kB/s | 46 kB 00:00 2023-12-11T20:04:11.843 INFO:teuthology.orchestra.run.smithi086.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2023-12-11T20:04:11.899 INFO:teuthology.orchestra.run.smithi086.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 294 kB/s | 16 kB 00:00 2023-12-11T20:04:11.925 INFO:teuthology.orchestra.run.smithi086.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 456 kB/s | 79 kB 00:00 2023-12-11T20:04:11.958 INFO:teuthology.orchestra.run.smithi086.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-12-11T20:04:12.044 INFO:teuthology.orchestra.run.smithi086.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 300 kB/s | 36 kB 00:00 2023-12-11T20:04:12.075 INFO:teuthology.orchestra.run.smithi086.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2023-12-11T20:04:12.134 INFO:teuthology.orchestra.run.smithi086.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2023-12-11T20:04:12.202 INFO:teuthology.orchestra.run.smithi086.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.7 MB/s | 427 kB 00:00 2023-12-11T20:04:12.759 INFO:teuthology.orchestra.run.smithi086.stdout:(36/36): ceph-mon-19.0.0-83.gc56ad3ea.el9.x86_6 1.6 MB/s | 4.4 MB 00:02 2023-12-11T20:04:12.763 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:12.763 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.6 MB/s | 30 MB 00:03 2023-12-11T20:04:12.776 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:12.818 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:12.818 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:13.103 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/6 2023-12-11T20:04:13.103 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/6 2023-12-11T20:04:13.104 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/6 2023-12-11T20:04:13.104 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-11T20:04:13.104 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-11T20:04:13.104 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-11T20:04:13.235 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:13.235 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:13.474 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-11T20:04:13.474 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:13.475 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:04:13.475 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 jq-1.6-16.el9.x86_64 2023-12-11T20:04:13.475 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2023-12-11T20:04:13.475 INFO:teuthology.orchestra.run.smithi133.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2023-12-11T20:04:13.475 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:13.475 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:13.785 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph 2023-12-11T20:04:14.369 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:53 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:14.518 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout: ceph x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 6.5 k 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 2.1 M 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.5 M 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 243 k 2023-12-11T20:04:14.521 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 4.4 M 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 18 M 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-11T20:04:14.522 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-11T20:04:14.523 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-11T20:04:14.523 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-11T20:04:14.523 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-11T20:04:14.525 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-11T20:04:14.525 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-11T20:04:14.525 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-11T20:04:14.525 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-11T20:04:14.525 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-11T20:04:14.525 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-11T20:04:14.525 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-11T20:04:14.525 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout:Install 36 Packages 2023-12-11T20:04:14.526 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:14.527 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 30 M 2023-12-11T20:04:14.527 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 112 M 2023-12-11T20:04:14.527 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:04:14.580 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:14.645 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-11T20:04:14.679 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-11T20:04:14.706 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-11T20:04:14.775 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-11T20:04:14.812 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-11T20:04:14.846 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-11T20:04:14.937 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-11T20:04:15.023 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-11T20:04:15.065 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-11T20:04:15.138 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-11T20:04:15.189 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-11T20:04:15.229 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-11T20:04:15.261 INFO:teuthology.orchestra.run.smithi133.stdout:(1/36): ceph-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 21 kB/s | 6.5 kB 00:00 2023-12-11T20:04:15.277 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-11T20:04:15.332 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-11T20:04:15.517 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-11T20:04:15.574 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-11T20:04:15.630 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-11T20:04:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-11T20:04:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-11T20:04:15.770 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-11T20:04:15.805 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-11T20:04:15.838 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-11T20:04:15.878 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-11T20:04:15.913 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-11T20:04:15.936 INFO:teuthology.orchestra.run.smithi133.stdout:(2/36): ceph-mgr-19.0.0-83.gc56ad3ea.el9.x86_64 1.5 MB/s | 1.5 MB 00:00 2023-12-11T20:04:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-11T20:04:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-11T20:04:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-11T20:04:16.141 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-11T20:04:16.279 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-11T20:04:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-11T20:04:16.569 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.no 31/36 2023-12-11T20:04:16.592 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 32/36 2023-12-11T20:04:16.595 INFO:teuthology.orchestra.run.smithi133.stdout:(3/36): ceph-mds-19.0.0-83.gc56ad3ea.el9.x86_64 1.3 MB/s | 2.1 MB 00:01 2023-12-11T20:04:16.617 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 32/36 2023-12-11T20:04:16.617 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:16.617 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-11T20:04:16.617 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-11T20:04:16.617 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-11T20:04:16.617 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:16.766 INFO:teuthology.orchestra.run.smithi133.stdout:(4/36): ceph-mgr-modules-core-19.0.0-83.gc56ad3 1.4 MB/s | 243 kB 00:00 2023-12-11T20:04:16.870 INFO:teuthology.orchestra.run.smithi133.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-12-11T20:04:16.912 INFO:teuthology.orchestra.run.smithi133.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-11T20:04:16.938 INFO:teuthology.orchestra.run.smithi133.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-12-11T20:04:16.963 INFO:teuthology.orchestra.run.smithi133.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2023-12-11T20:04:16.988 INFO:teuthology.orchestra.run.smithi133.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-12-11T20:04:17.030 INFO:teuthology.orchestra.run.smithi133.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-11T20:04:17.088 INFO:teuthology.orchestra.run.smithi133.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 20 MB/s | 1.1 MB 00:00 2023-12-11T20:04:17.121 INFO:teuthology.orchestra.run.smithi133.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-11T20:04:17.155 INFO:teuthology.orchestra.run.smithi133.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2023-12-11T20:04:17.189 INFO:teuthology.orchestra.run.smithi133.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-11T20:04:17.222 INFO:teuthology.orchestra.run.smithi133.stdout:(15/36): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-11T20:04:17.306 INFO:teuthology.orchestra.run.smithi133.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2023-12-11T20:04:17.806 INFO:teuthology.orchestra.run.smithi133.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 302 kB/s | 151 kB 00:00 2023-12-11T20:04:17.923 INFO:teuthology.orchestra.run.smithi133.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 373 kB/s | 43 kB 00:00 2023-12-11T20:04:18.082 INFO:teuthology.orchestra.run.smithi133.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2023-12-11T20:04:18.134 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 33/36 2023-12-11T20:04:18.160 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 33/36 2023-12-11T20:04:18.161 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:18.161 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-11T20:04:18.161 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-11T20:04:18.161 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-11T20:04:18.161 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:18.282 INFO:teuthology.orchestra.run.smithi133.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2023-12-11T20:04:18.374 INFO:teuthology.orchestra.run.smithi133.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2023-12-11T20:04:18.491 INFO:teuthology.orchestra.run.smithi133.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2023-12-11T20:04:18.591 INFO:teuthology.orchestra.run.smithi133.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2023-12-11T20:04:18.601 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 34/36 2023-12-11T20:04:18.624 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 34/36 2023-12-11T20:04:18.624 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:18.625 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-11T20:04:18.625 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-11T20:04:18.625 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-11T20:04:18.625 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:18.708 INFO:teuthology.orchestra.run.smithi133.stdout:(24/36): ceph-mon-19.0.0-83.gc56ad3ea.el9.x86_6 1.3 MB/s | 4.4 MB 00:03 2023-12-11T20:04:18.733 INFO:teuthology.orchestra.run.smithi133.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 137 kB/s | 19 kB 00:00 2023-12-11T20:04:18.834 INFO:teuthology.orchestra.run.smithi133.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2023-12-11T20:04:18.835 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 35/36 2023-12-11T20:04:18.858 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 35/36 2023-12-11T20:04:18.859 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:18.859 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-11T20:04:18.859 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-11T20:04:18.859 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-11T20:04:18.859 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:18.926 INFO:teuthology.orchestra.run.smithi133.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 859 kB/s | 79 kB 00:00 2023-12-11T20:04:18.926 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 36/36 2023-12-11T20:04:18.951 INFO:teuthology.orchestra.run.smithi133.stdout:(28/36): python3-jaraco-text-3.2.0-6.el9.noarch 81 kB/s | 20 kB 00:00 2023-12-11T20:04:19.051 INFO:teuthology.orchestra.run.smithi133.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2023-12-11T20:04:19.077 INFO:teuthology.orchestra.run.smithi133.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2023-12-11T20:04:19.285 INFO:teuthology.orchestra.run.smithi133.stdout:(31/36): ceph-osd-19.0.0-83.gc56ad3ea.el9.x86_6 5.3 MB/s | 18 MB 00:03 2023-12-11T20:04:19.302 INFO:teuthology.orchestra.run.smithi133.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 159 kB/s | 36 kB 00:00 2023-12-11T20:04:19.361 INFO:teuthology.orchestra.run.smithi133.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 291 kB/s | 90 kB 00:00 2023-12-11T20:04:19.461 INFO:teuthology.orchestra.run.smithi133.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2023-12-11T20:04:19.670 INFO:teuthology.orchestra.run.smithi133.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.1 MB/s | 427 kB 00:00 2023-12-11T20:04:19.795 INFO:teuthology.orchestra.run.smithi133.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2023-12-11T20:04:19.798 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:19.798 INFO:teuthology.orchestra.run.smithi133.stdout:Total 5.8 MB/s | 30 MB 00:05 2023-12-11T20:04:19.811 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:04:19.855 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:04:19.855 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:04:20.274 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:04:20.275 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:04:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 36/36 2023-12-11T20:04:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/36 2023-12-11T20:04:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/36 2023-12-11T20:04:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 3/36 2023-12-11T20:04:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 4/36 2023-12-11T20:04:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/36 2023-12-11T20:04:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.no 6/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-11T20:04:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-11T20:04:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-11T20:04:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-11T20:04:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-11T20:04:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-11T20:04:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-11T20:04:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-11T20:04:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-11T20:04:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-11T20:04:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-11T20:04:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-11T20:04:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-11T20:04:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-11T20:04:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-11T20:04:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-11T20:04:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-11T20:04:21.617 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:04:21.681 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-11T20:04:21.682 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-11T20:04:21.682 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:21.682 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:21.682 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: lua-5.4.4-4.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-11T20:04:21.683 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-11T20:04:21.684 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:21.685 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:21.716 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-11T20:04:21.758 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-11T20:04:21.830 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-11T20:04:21.934 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-11T20:04:22.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2023-12-11T20:04:22.065 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-11T20:04:22.156 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-11T20:04:22.234 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-11T20:04:22.277 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-11T20:04:22.347 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-11T20:04:22.401 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-11T20:04:22.441 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-11T20:04:22.498 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-11T20:04:22.565 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-11T20:04:22.641 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:57 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:22.734 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:04:22.752 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-11T20:04:22.783 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:22.784 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:04:22.784 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:22.802 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-11T20:04:22.841 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-11T20:04:22.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2023-12-11T20:04:22.875 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-11T20:04:22.910 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-11T20:04:22.982 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-11T20:04:23.016 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-11T20:04:23.049 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-11T20:04:23.090 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-11T20:04:23.125 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-11T20:04:23.167 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-11T20:04:23.257 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-11T20:04:23.315 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-11T20:04:23.386 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-11T20:04:23.413 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:58 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:23.523 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-11T20:04:23.557 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:23.557 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:04:23.557 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 802 k 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 802 k 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 817 k 2023-12-11T20:04:23.558 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:23.642 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-11T20:04:23.806 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.no 31/36 2023-12-11T20:04:23.828 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 32/36 2023-12-11T20:04:23.854 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 32/36 2023-12-11T20:04:23.854 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:23.854 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-11T20:04:23.854 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-11T20:04:23.854 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-11T20:04:23.854 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:24.900 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-19.0.0-83.gc56ad3ea.el9.noarch.rpm 598 kB/s | 802 kB 00:01 2023-12-11T20:04:24.901 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:24.901 INFO:teuthology.orchestra.run.smithi086.stdout:Total 597 kB/s | 802 kB 00:01 2023-12-11T20:04:24.901 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:24.907 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:24.907 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:24.913 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:24.914 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:24.986 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:25.371 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 33/36 2023-12-11T20:04:25.397 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 33/36 2023-12-11T20:04:25.397 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:25.397 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-11T20:04:25.397 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-11T20:04:25.398 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-11T20:04:25.398 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:25.424 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:25.838 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 34/36 2023-12-11T20:04:25.861 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 34/36 2023-12-11T20:04:25.861 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:25.861 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-11T20:04:25.861 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-11T20:04:25.861 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-11T20:04:25.861 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:25.896 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:26.063 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 35/36 2023-12-11T20:04:26.087 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 35/36 2023-12-11T20:04:26.087 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:26.087 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-11T20:04:26.087 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-11T20:04:26.087 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-11T20:04:26.087 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:26.152 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 36/36 2023-12-11T20:04:26.241 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:26.241 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:26.241 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:26.241 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:26.241 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:26.241 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:26.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2023-12-11T20:04:27.022 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:02 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:27.170 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 145 k 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:04:27.171 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:27.172 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 145 k 2023-12-11T20:04:27.172 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 440 k 2023-12-11T20:04:27.172 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:27.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-19.0.0-83.gc56ad3ea 671 kB/s | 145 kB 00:00 2023-12-11T20:04:27.389 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:27.389 INFO:teuthology.orchestra.run.smithi086.stdout:Total 667 kB/s | 145 kB 00:00 2023-12-11T20:04:27.389 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:27.396 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:27.397 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:27.448 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:27.449 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:27.601 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:27.673 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-11T20:04:27.697 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-11T20:04:27.697 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:27.697 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-11T20:04:27.697 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 36/36 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/36 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/36 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 3/36 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 4/36 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/36 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.no 6/36 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-11T20:04:27.838 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-11T20:04:27.839 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-11T20:04:27.840 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-11T20:04:27.840 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-11T20:04:27.840 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-11T20:04:27.840 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-11T20:04:27.840 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-11T20:04:27.840 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-11T20:04:27.841 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-11T20:04:27.841 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-11T20:04:27.841 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-11T20:04:27.841 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-11T20:04:27.841 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-11T20:04:27.841 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-11T20:04:27.841 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-11T20:04:28.656 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-11T20:04:28.656 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:28.656 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: lua-5.4.4-4.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-11T20:04:28.657 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply-3.11-14.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-11T20:04:28.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:28.659 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:28.691 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-11T20:04:28.691 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:28.691 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:28.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:28.691 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:28.691 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:28.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2023-12-11T20:04:28.927 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base 2023-12-11T20:04:29.489 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:04 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:29.505 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:08 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:29.584 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:04:29.600 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:04:29.633 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:29.634 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:04:29.634 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:29.649 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:29.650 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:04:29.650 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:29.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2023-12-11T20:04:29.760 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install cephadm 2023-12-11T20:04:30.326 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:05 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:30.336 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:09 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:30.470 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:30.471 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:30.471 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:04:30.471 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:30.471 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:30.471 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 1.8 M 2023-12-11T20:04:30.471 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 28 k 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 15 k 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout:Install 7 Packages 2023-12-11T20:04:30.472 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:30.473 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.1 M 2023-12-11T20:04:30.473 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 8.7 M 2023-12-11T20:04:30.473 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:30.480 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 802 k 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:04:30.481 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:30.482 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 802 k 2023-12-11T20:04:30.482 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 817 k 2023-12-11T20:04:30.482 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:04:30.716 INFO:teuthology.orchestra.run.smithi086.stdout:(1/7): ceph-prometheus-alerts-19.0.0-83.gc56ad3 79 kB/s | 15 kB 00:00 2023-12-11T20:04:30.758 INFO:teuthology.orchestra.run.smithi086.stdout:(2/7): ceph-grafana-dashboards-19.0.0-83.gc56ad 121 kB/s | 28 kB 00:00 2023-12-11T20:04:30.895 INFO:teuthology.orchestra.run.smithi133.stdout:cephadm-19.0.0-83.gc56ad3ea.el9.noarch.rpm 1.9 MB/s | 802 kB 00:00 2023-12-11T20:04:30.896 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:30.896 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.9 MB/s | 802 kB 00:00 2023-12-11T20:04:30.896 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:04:30.901 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:04:30.901 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:04:30.907 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:04:30.907 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:04:30.908 INFO:teuthology.orchestra.run.smithi086.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2023-12-11T20:04:30.942 INFO:teuthology.orchestra.run.smithi086.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 254 kB/s | 41 kB 00:00 2023-12-11T20:04:30.975 INFO:teuthology.orchestra.run.smithi086.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 460 kB/s | 31 kB 00:00 2023-12-11T20:04:30.984 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:04:31.059 INFO:teuthology.orchestra.run.smithi086.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2023-12-11T20:04:31.118 INFO:teuthology.orchestra.run.smithi086.stdout:(7/7): ceph-mgr-dashboard-19.0.0-83.gc56ad3ea.e 3.0 MB/s | 1.8 MB 00:00 2023-12-11T20:04:31.120 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:31.121 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.2 MB/s | 2.1 MB 00:00 2023-12-11T20:04:31.121 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:31.129 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:31.129 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:31.188 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:31.188 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:31.383 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:31.475 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:31.483 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:31.552 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-11T20:04:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-11T20:04:31.640 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-11T20:04:31.672 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-11T20:04:31.703 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noa 5/7 2023-12-11T20:04:31.854 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:31.939 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.no 6/7 2023-12-11T20:04:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 7/7 2023-12-11T20:04:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:32.148 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:04:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:32.148 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:32.310 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-immutable-object-cache 2023-12-11T20:04:32.488 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 7/7 2023-12-11T20:04:32.489 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.no 1/7 2023-12-11T20:04:32.489 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 2/7 2023-12-11T20:04:32.489 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noa 3/7 2023-12-11T20:04:32.489 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-11T20:04:32.489 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-11T20:04:32.489 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-11T20:04:32.806 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-11T20:04:32.806 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:32.806 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:32.807 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:32.898 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:11 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:33.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-11T20:04:33.048 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:33.048 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 145 k 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:04:33.049 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:33.050 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 145 k 2023-12-11T20:04:33.050 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 440 k 2023-12-11T20:04:33.050 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:04:33.267 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-immutable-object-cache-19.0.0-83.gc56ad3ea 670 kB/s | 145 kB 00:00 2023-12-11T20:04:33.268 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:33.268 INFO:teuthology.orchestra.run.smithi133.stdout:Total 666 kB/s | 145 kB 00:00 2023-12-11T20:04:33.268 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:04:33.275 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:04:33.275 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:04:33.327 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:04:33.327 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:04:33.475 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:04:33.551 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-11T20:04:33.576 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-11T20:04:33.576 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:33.576 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-11T20:04:33.576 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:33.601 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:08 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:33.747 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:33.748 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================== 2023-12-11T20:04:33.748 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:04:33.748 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================== 2023-12-11T20:04:33.748 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:33.748 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 7.4 M 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:33.749 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:33.750 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================== 2023-12-11T20:04:33.750 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2023-12-11T20:04:33.750 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:33.750 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2023-12-11T20:04:33.750 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 209 M 2023-12-11T20:04:33.750 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:33.924 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2023-12-11T20:04:33.958 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2023-12-11T20:04:34.016 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2023-12-11T20:04:34.042 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 418 kB/s | 34 kB 00:00 2023-12-11T20:04:34.159 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 33 MB/s | 4.7 MB 00:00 2023-12-11T20:04:34.226 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-12-11T20:04:34.250 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-11T20:04:34.309 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-12-11T20:04:34.460 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2023-12-11T20:04:34.485 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-11T20:04:34.592 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-11T20:04:34.593 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:34.593 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:04:34.593 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:04:34.593 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:34.593 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:34.794 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr 2023-12-11T20:04:35.093 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 388 kB/s | 245 kB 00:00 2023-12-11T20:04:35.386 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:14 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:35.481 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:04:35.531 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:35.532 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:04:35.532 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:35.577 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-8 4.1 MB/s | 7.4 MB 00:01 2023-12-11T20:04:35.581 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:35.581 INFO:teuthology.orchestra.run.smithi086.stdout:Total 20 MB/s | 37 MB 00:01 2023-12-11T20:04:35.594 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:35.595 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-dashboard 2023-12-11T20:04:35.619 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:35.619 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:35.992 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:35.992 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:36.167 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:15 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:36.311 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:36.312 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:36.312 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:04:36.312 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:36.312 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:04:36.312 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 1.8 M 2023-12-11T20:04:36.312 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 28 k 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 15 k 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout:Installing weak dependencies: 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout:Install 7 Packages 2023-12-11T20:04:36.313 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:36.314 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 2.1 M 2023-12-11T20:04:36.314 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 8.7 M 2023-12-11T20:04:36.314 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:04:36.691 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:36.783 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-11T20:04:36.823 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-11T20:04:36.920 INFO:teuthology.orchestra.run.smithi133.stdout:(1/7): ceph-grafana-dashboards-19.0.0-83.gc56ad 118 kB/s | 28 kB 00:00 2023-12-11T20:04:36.924 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-11T20:04:36.945 INFO:teuthology.orchestra.run.smithi133.stdout:(2/7): ceph-prometheus-alerts-19.0.0-83.gc56ad3 57 kB/s | 15 kB 00:00 2023-12-11T20:04:36.976 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-11T20:04:37.154 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-11T20:04:37.162 INFO:teuthology.orchestra.run.smithi133.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 37 kB/s | 9.0 kB 00:00 2023-12-11T20:04:37.183 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-11T20:04:37.263 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-11T20:04:37.354 INFO:teuthology.orchestra.run.smithi133.stdout:(4/7): ceph-mgr-dashboard-19.0.0-83.gc56ad3ea.e 2.6 MB/s | 1.8 MB 00:00 2023-12-11T20:04:37.371 INFO:teuthology.orchestra.run.smithi133.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 96 kB/s | 41 kB 00:00 2023-12-11T20:04:37.388 INFO:teuthology.orchestra.run.smithi133.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 136 kB/s | 31 kB 00:00 2023-12-11T20:04:37.663 INFO:teuthology.orchestra.run.smithi133.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 611 kB/s | 188 kB 00:00 2023-12-11T20:04:37.665 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:37.665 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.5 MB/s | 2.1 MB 00:01 2023-12-11T20:04:37.666 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:04:37.673 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:04:37.673 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:04:37.740 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:04:37.740 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:04:37.908 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-11T20:04:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-11T20:04:38.049 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:04:38.106 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-11T20:04:38.148 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-11T20:04:38.194 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-11T20:04:38.226 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-11T20:04:38.248 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noa 5/7 2023-12-11T20:04:38.475 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.no 6/7 2023-12-11T20:04:38.509 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 7/7 2023-12-11T20:04:39.063 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 7/7 2023-12-11T20:04:39.063 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.no 1/7 2023-12-11T20:04:39.063 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 2/7 2023-12-11T20:04:39.063 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noa 3/7 2023-12-11T20:04:39.063 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-11T20:04:39.063 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-11T20:04:39.063 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-11T20:04:39.229 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-11T20:04:39.382 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:39.383 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:39.625 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-11T20:04:40.205 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:19 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:40.255 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-11T20:04:40.282 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 12/12 2023-12-11T20:04:40.352 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:40.353 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================== 2023-12-11T20:04:40.353 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:04:40.353 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================== 2023-12-11T20:04:40.353 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:04:40.353 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 7.4 M 2023-12-11T20:04:40.353 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-11T20:04:40.354 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-11T20:04:40.355 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:40.355 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:04:40.355 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================== 2023-12-11T20:04:40.355 INFO:teuthology.orchestra.run.smithi133.stdout:Install 12 Packages 2023-12-11T20:04:40.355 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:40.355 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 37 M 2023-12-11T20:04:40.355 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 209 M 2023-12-11T20:04:40.356 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:04:40.536 INFO:teuthology.orchestra.run.smithi133.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 224 kB/s | 32 kB 00:00 2023-12-11T20:04:40.570 INFO:teuthology.orchestra.run.smithi133.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 522 kB/s | 17 kB 00:00 2023-12-11T20:04:40.629 INFO:teuthology.orchestra.run.smithi133.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2023-12-11T20:04:40.654 INFO:teuthology.orchestra.run.smithi133.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 420 kB/s | 34 kB 00:00 2023-12-11T20:04:40.771 INFO:teuthology.orchestra.run.smithi133.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 33 MB/s | 4.7 MB 00:00 2023-12-11T20:04:40.846 INFO:teuthology.orchestra.run.smithi133.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2023-12-11T20:04:40.871 INFO:teuthology.orchestra.run.smithi133.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-12-11T20:04:40.938 INFO:teuthology.orchestra.run.smithi133.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-12-11T20:04:41.097 INFO:teuthology.orchestra.run.smithi133.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2023-12-11T20:04:41.122 INFO:teuthology.orchestra.run.smithi133.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-11T20:04:41.314 INFO:teuthology.orchestra.run.smithi133.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-8 8.0 MB/s | 7.4 MB 00:00 2023-12-11T20:04:41.339 INFO:teuthology.orchestra.run.smithi133.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2023-12-11T20:04:41.341 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:41.342 INFO:teuthology.orchestra.run.smithi133.stdout:Total 38 MB/s | 37 MB 00:00 2023-12-11T20:04:41.354 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:04:41.378 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:04:41.378 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:04:41.753 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:04:41.753 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:04:42.452 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:04:42.514 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-11T20:04:42.543 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-11T20:04:42.651 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-11T20:04:42.688 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 12/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 1/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-11T20:04:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-11T20:04:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-11T20:04:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-11T20:04:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-11T20:04:42.853 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-11T20:04:42.887 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-11T20:04:42.973 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-11T20:04:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-11T20:04:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:43.192 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:43.193 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:43.583 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2023-12-11T20:04:43.613 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-11T20:04:43.714 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-11T20:04:44.167 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:19 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:44.313 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 50 k 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-11T20:04:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2023-12-11T20:04:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:44.316 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2023-12-11T20:04:44.316 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2023-12-11T20:04:44.316 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:44.523 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 155 kB/s | 18 kB 00:00 2023-12-11T20:04:44.548 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 172 kB/s | 24 kB 00:00 2023-12-11T20:04:44.590 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-11T20:04:44.616 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-11T20:04:44.641 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): ceph-mgr-rook-19.0.0-83.gc56ad3ea.el9.n 214 kB/s | 50 kB 00:00 2023-12-11T20:04:44.666 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2023-12-11T20:04:44.708 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 466 kB/s | 43 kB 00:00 2023-12-11T20:04:44.825 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 95 kB/s | 15 kB 00:00 2023-12-11T20:04:44.850 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2023-12-11T20:04:44.938 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-11T20:04:44.942 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 844 kB/s | 197 kB 00:00 2023-12-11T20:04:44.967 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2023-12-11T20:04:45.009 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2023-12-11T20:04:45.109 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2023-12-11T20:04:45.111 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:45.111 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2023-12-11T20:04:45.112 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:45.124 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:45.124 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:45.221 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:45.221 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:45.728 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:45.895 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-11T20:04:45.951 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-11T20:04:45.967 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-11T20:04:45.994 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 12/12 2023-12-11T20:04:46.007 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-11T20:04:46.044 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-11T20:04:46.085 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-11T20:04:46.177 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-11T20:04:46.287 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-11T20:04:46.337 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-11T20:04:47.071 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-11T20:04:47.117 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-11T20:04:47.151 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-11T20:04:47.198 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-11T20:04:47.221 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 13/13 2023-12-11T20:04:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 13/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 1/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-11T20:04:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-11T20:04:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-11T20:04:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:48.405 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-11T20:04:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-11T20:04:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-11T20:04:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-11T20:04:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-11T20:04:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-11T20:04:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-11T20:04:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-11T20:04:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-11T20:04:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-11T20:04:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-11T20:04:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-11T20:04:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:48.407 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:48.596 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 12/12 2023-12-11T20:04:48.596 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 1/12 2023-12-11T20:04:48.596 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-11T20:04:48.596 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-11T20:04:48.597 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-11T20:04:48.597 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-11T20:04:48.597 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-11T20:04:48.597 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-11T20:04:48.597 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-11T20:04:48.597 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-11T20:04:48.597 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-11T20:04:48.597 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-11T20:04:48.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2023-12-11T20:04:49.055 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-11T20:04:49.055 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:49.055 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:04:49.055 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:49.055 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-11T20:04:49.055 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-11T20:04:49.055 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-11T20:04:49.055 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:49.056 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:49.231 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:24 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:49.377 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:49.378 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:04:49.378 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:04:49.378 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:04:49.378 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 133 k 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-12-11T20:04:49.379 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:49.380 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.8 M 2023-12-11T20:04:49.380 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2023-12-11T20:04:49.380 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:49.476 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-rook 2023-12-11T20:04:49.618 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-11T20:04:49.735 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-12-11T20:04:49.860 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): ceph-mgr-cephadm-19.0.0-83.gc56ad3ea.el9 333 kB/s | 133 kB 00:00 2023-12-11T20:04:50.035 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 208 kB/s | 58 kB 00:00 2023-12-11T20:04:50.063 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:29 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:50.127 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2023-12-11T20:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2023-12-11T20:04:50.214 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 50 k 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-11T20:04:50.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout:Install 13 Packages 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 2.0 M 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 26 M 2023-12-11T20:04:50.216 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.7 MB/s | 6.8 MB 00:00 2023-12-11T20:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:50.387 INFO:teuthology.orchestra.run.smithi133.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 203 kB/s | 24 kB 00:00 2023-12-11T20:04:50.412 INFO:teuthology.orchestra.run.smithi133.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2023-12-11T20:04:50.454 INFO:teuthology.orchestra.run.smithi133.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-11T20:04:50.479 INFO:teuthology.orchestra.run.smithi133.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-11T20:04:50.504 INFO:teuthology.orchestra.run.smithi133.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-11T20:04:50.530 INFO:teuthology.orchestra.run.smithi133.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 861 kB/s | 43 kB 00:00 2023-12-11T20:04:50.546 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:50.563 INFO:teuthology.orchestra.run.smithi133.stdout:(7/13): ceph-mgr-rook-19.0.0-83.gc56ad3ea.el9.n 169 kB/s | 50 kB 00:00 2023-12-11T20:04:50.689 INFO:teuthology.orchestra.run.smithi133.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2023-12-11T20:04:50.714 INFO:teuthology.orchestra.run.smithi133.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 81 kB/s | 15 kB 00:00 2023-12-11T20:04:50.726 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-11T20:04:50.778 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-11T20:04:50.806 INFO:teuthology.orchestra.run.smithi133.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 645 kB/s | 59 kB 00:00 2023-12-11T20:04:50.831 INFO:teuthology.orchestra.run.smithi133.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 890 kB/s | 197 kB 00:00 2023-12-11T20:04:50.856 INFO:teuthology.orchestra.run.smithi133.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2023-12-11T20:04:50.956 INFO:teuthology.orchestra.run.smithi133.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2023-12-11T20:04:50.960 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:50.960 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2023-12-11T20:04:50.961 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:04:50.973 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:04:50.973 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:04:51.069 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:04:51.069 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:04:51.299 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-11T20:04:51.381 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-11T20:04:51.442 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-11T20:04:51.466 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 6/6 2023-12-11T20:04:51.588 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:04:51.744 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-11T20:04:51.791 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-11T20:04:51.846 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-11T20:04:51.883 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-11T20:04:51.925 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-11T20:04:52.018 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-11T20:04:52.118 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-11T20:04:52.168 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-11T20:04:52.226 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 6/6 2023-12-11T20:04:52.226 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/6 2023-12-11T20:04:52.226 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-11T20:04:52.226 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-11T20:04:52.226 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-11T20:04:52.226 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-11T20:04:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-11T20:04:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:52.547 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:52.548 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:52.548 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-11T20:04:52.548 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-11T20:04:52.548 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-11T20:04:52.548 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-11T20:04:52.548 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-11T20:04:52.548 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:52.548 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:52.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2023-12-11T20:04:52.891 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-11T20:04:52.940 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-11T20:04:52.974 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-11T20:04:53.021 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-11T20:04:53.044 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 13/13 2023-12-11T20:04:53.388 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:28 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:53.536 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:53.536 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:53.536 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:04:53.536 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:53.536 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:53.537 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 884 k 2023-12-11T20:04:53.537 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:04:53.537 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-11T20:04:53.537 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:53.537 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:53.537 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:53.537 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2023-12-11T20:04:53.537 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:53.538 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 963 k 2023-12-11T20:04:53.538 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.8 M 2023-12-11T20:04:53.538 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:53.654 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 681 kB/s | 79 kB 00:00 2023-12-11T20:04:53.825 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 13/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 1/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-11T20:04:53.826 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-11T20:04:53.845 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-19.0.0-83.gc56ad3ea.el9.x86_64 2.8 MB/s | 884 kB 00:00 2023-12-11T20:04:53.846 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:53.847 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.0 MB/s | 963 kB 00:00 2023-12-11T20:04:53.847 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:53.867 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:53.867 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:53.943 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:53.943 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:54.137 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:54.194 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-11T20:04:54.195 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-11T20:04:54.196 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-11T20:04:54.196 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-11T20:04:54.196 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-11T20:04:54.196 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-11T20:04:54.196 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:54.196 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:54.279 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-11T20:04:54.353 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:04:54.455 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-cephadm 2023-12-11T20:04:55.044 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:34 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:55.070 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:04:55.070 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-11T20:04:55.189 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:55.190 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:04:55.190 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:04:55.190 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:04:55.190 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:04:55.190 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 133 k 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2023-12-11T20:04:55.191 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:55.192 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 6.8 M 2023-12-11T20:04:55.192 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 33 M 2023-12-11T20:04:55.192 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:04:55.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-11T20:04:55.380 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:55.380 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:55.380 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-11T20:04:55.380 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:55.380 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:55.508 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-12-11T20:04:55.608 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2023-12-11T20:04:55.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-volume 2023-12-11T20:04:55.717 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): ceph-mgr-cephadm-19.0.0-83.gc56ad3ea.el9 369 kB/s | 133 kB 00:00 2023-12-11T20:04:55.917 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2023-12-11T20:04:56.025 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2023-12-11T20:04:56.051 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2023-12-11T20:04:56.053 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:56.053 INFO:teuthology.orchestra.run.smithi133.stdout:Total 7.9 MB/s | 6.8 MB 00:00 2023-12-11T20:04:56.053 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:04:56.063 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:04:56.063 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:04:56.152 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:04:56.152 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:04:56.252 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:31 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:56.396 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:56.397 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 262 k 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 262 k 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.2 M 2023-12-11T20:04:56.398 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:56.477 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:04:56.632 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-11T20:04:56.685 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-11T20:04:56.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-volume-19.0.0-83.gc56ad3ea.el9.noarch.rpm 665 kB/s | 262 kB 00:00 2023-12-11T20:04:56.793 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:56.794 INFO:teuthology.orchestra.run.smithi086.stdout:Total 663 kB/s | 262 kB 00:00 2023-12-11T20:04:56.794 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:56.801 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:56.801 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:56.834 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:56.834 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:57.111 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:57.127 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:57.127 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:04:57.127 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-11T20:04:57.127 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:57.227 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-11T20:04:57.313 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-11T20:04:57.373 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-11T20:04:57.398 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 6/6 2023-12-11T20:04:58.131 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:04:58.131 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:58.131 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:04:58.131 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:58.131 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:58.131 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:04:58.236 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 6/6 2023-12-11T20:04:58.236 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/6 2023-12-11T20:04:58.236 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-11T20:04:58.236 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-11T20:04:58.236 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-11T20:04:58.236 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-11T20:04:58.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2023-12-11T20:04:58.547 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-11T20:04:58.547 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:58.547 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:04:58.547 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:04:58.547 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-11T20:04:58.548 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-11T20:04:58.548 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-11T20:04:58.548 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-11T20:04:58.548 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-11T20:04:58.548 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:58.548 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:04:58.809 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-fuse 2023-12-11T20:04:59.026 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:34 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:04:59.173 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 126 k 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:04:59.174 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:04:59.175 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 126 k 2023-12-11T20:04:59.175 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 456 k 2023-12-11T20:04:59.175 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:04:59.392 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:38 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:04:59.405 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-19.0.0-83.gc56ad3ea.el9.x86_64.r 550 kB/s | 126 kB 00:00 2023-12-11T20:04:59.406 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:59.406 INFO:teuthology.orchestra.run.smithi086.stdout:Total 546 kB/s | 126 kB 00:00 2023-12-11T20:04:59.406 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:04:59.411 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:04:59.411 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:04:59.466 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:04:59.466 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:04:59.539 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:04:59.539 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:59.539 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:04:59.539 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:59.539 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:04:59.540 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 884 k 2023-12-11T20:04:59.540 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:04:59.540 INFO:teuthology.orchestra.run.smithi133.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-11T20:04:59.540 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:59.540 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:04:59.540 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:04:59.540 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2023-12-11T20:04:59.540 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:04:59.541 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 963 k 2023-12-11T20:04:59.541 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.8 M 2023-12-11T20:04:59.541 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:04:59.618 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:04:59.661 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 656 kB/s | 79 kB 00:00 2023-12-11T20:04:59.728 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:04:59.986 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): ceph-fuse-19.0.0-83.gc56ad3ea.el9.x86_64 1.9 MB/s | 884 kB 00:00 2023-12-11T20:04:59.987 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:04:59.987 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.1 MB/s | 963 kB 00:00 2023-12-11T20:04:59.987 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:05:00.007 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:05:00.008 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:05:00.082 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:05:00.082 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:05:00.159 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:00.275 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:05:00.419 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-11T20:05:00.493 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:05:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:00.509 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:05:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:00.510 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:00.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2023-12-11T20:05:01.204 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:05:01.205 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-11T20:05:01.318 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:36 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:05:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:01.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2023-12-11T20:05:01.545 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-11T20:05:01.546 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:01.546 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:05:01.546 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-11T20:05:01.546 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:01.546 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:01.839 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-volume 2023-12-11T20:05:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:37 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 31 k 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:05:02.266 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 k 2023-12-11T20:05:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 139 k 2023-12-11T20:05:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:05:02.436 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-19.0.0-83.gc56ad3ea.el9.x86_64. 178 kB/s | 31 kB 00:00 2023-12-11T20:05:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:Total 177 kB/s | 31 kB 00:00 2023-12-11T20:05:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:05:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:05:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:05:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:05:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:05:02.575 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:05:02.581 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:02.581 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:02.581 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-12-11T20:05:02.581 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:02.582 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:05:02.582 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 262 k 2023-12-11T20:05:02.582 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:02.582 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:05:02.582 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:02.582 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:05:02.582 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:02.583 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 262 k 2023-12-11T20:05:02.583 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.2 M 2023-12-11T20:05:02.583 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:05:02.689 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:02.823 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-volume-19.0.0-83.gc56ad3ea.el9.noarch.rpm 1.1 MB/s | 262 kB 00:00 2023-12-11T20:05:02.824 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:02.824 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-12-11T20:05:02.824 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:05:02.831 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:05:02.831 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:05:02.867 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:05:02.867 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:05:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:03.085 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:05:03.143 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:05:03.159 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:05:03.159 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:05:03.159 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-11T20:05:03.159 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:03.347 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:03.347 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:05:03.347 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:03.347 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:03.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2023-12-11T20:05:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:39 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:04.160 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-11T20:05:04.160 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:04.161 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:05:04.161 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-11T20:05:04.161 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:04.161 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:05:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:04.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2023-12-11T20:05:04.455 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados-devel 2023-12-11T20:05:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:39 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:05:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:05.055 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:05.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2023-12-11T20:05:05.202 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:05.203 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:05.203 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:05:05.203 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 126 k 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 126 k 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 456 k 2023-12-11T20:05:05.204 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:05:05.562 INFO:teuthology.orchestra.run.smithi133.stdout:librados-devel-19.0.0-83.gc56ad3ea.el9.x86_64.r 354 kB/s | 126 kB 00:00 2023-12-11T20:05:05.564 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:05.565 INFO:teuthology.orchestra.run.smithi133.stdout:Total 352 kB/s | 126 kB 00:00 2023-12-11T20:05:05.565 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:05:05.569 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:05:05.569 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:05:05.625 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:05:05.626 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:05:05.685 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:40 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:05.780 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:05.786 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:05:05.829 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:05.830 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:05:05.830 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:05.894 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:05.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2023-12-11T20:05:06.373 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:06.487 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:06.582 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:06.632 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:06.633 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:05:06.633 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:06.685 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:06.686 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:06.686 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:05:06.686 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:06.686 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:06.686 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:06.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2023-12-11T20:05:06.956 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs2 2023-12-11T20:05:07.294 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:07.390 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:07.440 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:07.441 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:05:07.441 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:07.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2023-12-11T20:05:07.562 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:46 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:07.659 INFO:teuthology.orchestra.run.smithi133.stdout:Package libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:07.709 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:07.710 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:05:07.710 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:07.780 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs-devel 2023-12-11T20:05:08.104 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:08.198 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:08.248 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:08.249 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:05:08.249 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:08.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2023-12-11T20:05:08.362 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:47 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:08.507 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 31 k 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:05:08.508 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:08.509 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 31 k 2023-12-11T20:05:08.509 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 139 k 2023-12-11T20:05:08.509 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:05:08.665 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs-devel-19.0.0-83.gc56ad3ea.el9.x86_64. 199 kB/s | 31 kB 00:00 2023-12-11T20:05:08.666 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:08.666 INFO:teuthology.orchestra.run.smithi133.stdout:Total 198 kB/s | 31 kB 00:00 2023-12-11T20:05:08.666 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:05:08.669 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:05:08.670 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:05:08.700 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:05:08.700 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:05:08.780 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:05:08.887 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:08.897 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:09.045 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:09.046 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:09.046 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-12-11T20:05:09.046 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:09.046 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:05:09.046 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 85 k 2023-12-11T20:05:09.046 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:09.047 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:05:09.047 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:09.047 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:05:09.047 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:09.047 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 85 k 2023-12-11T20:05:09.047 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 231 k 2023-12-11T20:05:09.047 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:05:09.286 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:09.361 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 272 kB/s | 85 kB 00:00 2023-12-11T20:05:09.361 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:09.362 INFO:teuthology.orchestra.run.smithi086.stdout:Total 270 kB/s | 85 kB 00:00 2023-12-11T20:05:09.362 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:05:09.369 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:05:09.369 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:05:09.424 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:05:09.424 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:05:09.559 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:05:09.561 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:09.561 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:09.561 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:05:09.561 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:09.561 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:09.562 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:09.658 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:09.750 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados2 2023-12-11T20:05:10.085 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:10.334 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:49 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:10.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:10.399 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:10.399 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:05:10.399 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:10.399 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:10.399 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:10.430 INFO:teuthology.orchestra.run.smithi133.stdout:Package librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:10.481 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:10.482 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:05:10.482 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:10.557 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librbd1 2023-12-11T20:05:10.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2023-12-11T20:05:11.139 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:50 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:11.231 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:46 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:11.235 INFO:teuthology.orchestra.run.smithi133.stdout:Package librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:11.286 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:11.287 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:05:11.287 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:11.361 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rados 2023-12-11T20:05:11.389 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:11.389 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.1 M 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.1 M 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 13 M 2023-12-11T20:05:11.390 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:05:11.781 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 7.9 MB/s | 3.1 MB 00:00 2023-12-11T20:05:11.781 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:11.782 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2023-12-11T20:05:11.782 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:05:11.791 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:05:11.791 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:05:11.863 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:05:11.864 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:05:11.949 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:50 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:12.044 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:12.093 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:12.094 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:05:12.094 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:12.169 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rgw 2023-12-11T20:05:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:05:12.357 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:12.382 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:12.382 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:05:12.382 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-11T20:05:12.382 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-11T20:05:12.382 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-11T20:05:12.382 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:12.757 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:51 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:12.851 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:12.900 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:12.902 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:05:12.902 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:12.971 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-cephfs 2023-12-11T20:05:13.518 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:13.519 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:13.519 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:05:13.519 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:13.519 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:13.519 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:13.553 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:52 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:13.648 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:13.698 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:13.699 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:05:13.699 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:13.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2023-12-11T20:05:13.780 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rbd 2023-12-11T20:05:14.342 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:49 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:14.363 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:53 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:14.459 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:05:14.489 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:14.490 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:14.490 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-12-11T20:05:14.490 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:14.490 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:05:14.490 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 170 k 2023-12-11T20:05:14.490 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:14.491 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:05:14.491 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:14.491 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:05:14.491 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:14.491 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 170 k 2023-12-11T20:05:14.491 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 494 k 2023-12-11T20:05:14.491 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:05:14.508 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:14.509 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:05:14.509 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:14.585 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-fuse 2023-12-11T20:05:14.826 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 508 kB/s | 170 kB 00:00 2023-12-11T20:05:14.826 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:14.827 INFO:teuthology.orchestra.run.smithi086.stdout:Total 506 kB/s | 170 kB 00:00 2023-12-11T20:05:14.827 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:05:14.834 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:05:14.834 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:05:14.890 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:05:14.890 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:05:15.034 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:05:15.140 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:15.167 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:54 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:15.314 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:15.314 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:15.314 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 85 k 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 85 k 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 231 k 2023-12-11T20:05:15.315 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:05:15.526 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 404 kB/s | 85 kB 00:00 2023-12-11T20:05:15.527 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:15.527 INFO:teuthology.orchestra.run.smithi133.stdout:Total 402 kB/s | 85 kB 00:00 2023-12-11T20:05:15.527 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:05:15.534 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:05:15.535 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:05:15.574 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:15.590 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:05:15.590 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:05:15.724 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:05:15.818 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:15.894 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:15.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:15.894 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:05:15.894 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:15.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:15.894 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:16.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base-debuginfo 2023-12-11T20:05:16.246 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:16.574 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:16.574 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:16.574 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:05:16.575 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:16.575 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:16.575 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:16.705 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:51 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:16.797 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-mirror 2023-12-11T20:05:16.854 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 73 M 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 21 M 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:16.855 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2023-12-11T20:05:16.856 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:16.856 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 94 M 2023-12-11T20:05:16.856 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 513 M 2023-12-11T20:05:16.856 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:05:17.407 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:56 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.1 M 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:05:17.560 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:17.561 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:05:17.561 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:17.561 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 3.1 M 2023-12-11T20:05:17.561 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 13 M 2023-12-11T20:05:17.561 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:05:18.320 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-debuginfo-19.0.0-83.gc56ad3ea.el9.x 14 MB/s | 21 MB 00:01 2023-12-11T20:05:18.321 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-mirror-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 4.1 MB/s | 3.1 MB 00:00 2023-12-11T20:05:18.322 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:18.322 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.0 MB/s | 3.1 MB 00:00 2023-12-11T20:05:18.323 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:05:18.332 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:05:18.333 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:05:18.407 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:05:18.408 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:05:18.842 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:05:18.889 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:18.914 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:18.915 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-11T20:05:18.915 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-11T20:05:18.915 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-11T20:05:18.915 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-11T20:05:18.916 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:20.072 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:20.072 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:20.072 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:05:20.072 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:20.072 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:20.072 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:20.286 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-nbd 2023-12-11T20:05:20.878 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:59 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:21.026 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:21.027 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:21.027 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-12-11T20:05:21.027 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:21.027 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:05:21.027 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 170 k 2023-12-11T20:05:21.027 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:21.027 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:05:21.027 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:21.028 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:05:21.028 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:21.028 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 170 k 2023-12-11T20:05:21.028 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 494 k 2023-12-11T20:05:21.028 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:05:21.383 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-nbd-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 481 kB/s | 170 kB 00:00 2023-12-11T20:05:21.383 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:21.383 INFO:teuthology.orchestra.run.smithi133.stdout:Total 479 kB/s | 170 kB 00:00 2023-12-11T20:05:21.384 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:05:21.391 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:05:21.391 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:05:21.394 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-base-debuginfo-19.0.0-83.gc56ad3ea. 16 MB/s | 73 MB 00:04 2023-12-11T20:05:21.396 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:21.397 INFO:teuthology.orchestra.run.smithi086.stdout:Total 21 MB/s | 94 MB 00:04 2023-12-11T20:05:21.397 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:05:21.401 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:05:21.402 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:05:21.448 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:05:21.448 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:05:21.590 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:05:21.697 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:22.018 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:05:22.019 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:05:22.135 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:22.446 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:05:22.446 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:22.447 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:05:22.447 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:22.447 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:22.447 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:22.658 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base-debuginfo 2023-12-11T20:05:23.256 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:02 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:23.402 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:23.406 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:23.406 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:05:23.406 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:23.406 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:05:23.406 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 73 M 2023-12-11T20:05:23.406 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-12-11T20:05:23.406 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 21 M 2023-12-11T20:05:23.407 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:23.407 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:05:23.407 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:23.407 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2023-12-11T20:05:23.407 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:23.407 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 94 M 2023-12-11T20:05:23.407 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 513 M 2023-12-11T20:05:23.407 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:05:25.036 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:05:26.185 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): ceph-debuginfo-19.0.0-83.gc56ad3ea.el9.x 7.6 MB/s | 21 MB 00:02 2023-12-11T20:05:27.544 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): ceph-base-debuginfo-19.0.0-83.gc56ad3ea. 18 MB/s | 73 MB 00:04 2023-12-11T20:05:27.545 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:27.546 INFO:teuthology.orchestra.run.smithi133.stdout:Total 23 MB/s | 94 MB 00:04 2023-12-11T20:05:27.546 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:05:27.550 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:05:27.550 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:05:28.163 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:05:28.163 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:05:31.154 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:05:32.629 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-11T20:05:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:05:36.014 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:05:36.014 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-11T20:05:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:05:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:36.292 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:05:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:36.292 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:05:36.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-common-debuginfo 2023-12-11T20:05:37.087 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:12 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:05:37.233 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:05:37.233 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:05:37.233 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 404 M 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 404 M 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.6 G 2023-12-11T20:05:37.234 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:05:38.744 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-11T20:05:39.052 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:05:42.303 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:05:42.303 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-11T20:05:42.570 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-11T20:05:42.570 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:42.570 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:05:42.570 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:42.570 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:05:42.570 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:42.570 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:05:42.788 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-common-debuginfo 2023-12-11T20:05:43.376 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:22 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:05:43.522 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 404 M 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:05:43.523 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 404 M 2023-12-11T20:05:43.524 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.6 G 2023-12-11T20:05:43.524 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:05:55.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-common-debuginfo-19.0.0-83.gc56ad3ea.el9.x 22 MB/s | 404 MB 00:18 2023-12-11T20:05:55.816 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:05:55.816 INFO:teuthology.orchestra.run.smithi086.stdout:Total 22 MB/s | 404 MB 00:18 2023-12-11T20:05:55.816 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:05:55.820 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:05:55.820 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:05:58.233 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:05:58.233 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:06:27.800 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-common-debuginfo-19.0.0-83.gc56ad3ea.el9.x 9.1 MB/s | 404 MB 00:44 2023-12-11T20:06:27.801 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:06:27.801 INFO:teuthology.orchestra.run.smithi133.stdout:Total 9.1 MB/s | 404 MB 00:44 2023-12-11T20:06:27.802 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:06:27.806 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:06:27.806 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:06:30.217 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:06:30.217 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:06:38.972 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:06:39.085 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:06:47.554 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:06:47.831 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:06:47.831 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:06:47.831 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:06:47.832 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:06:47.832 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:06:47.832 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:06:48.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-11T20:06:48.662 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:23 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:06:48.817 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:06:48.817 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:06:48.817 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.4 M 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.4 M 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 6.4 M 2023-12-11T20:06:48.818 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:06:49.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-83 2.5 MB/s | 1.4 MB 00:00 2023-12-11T20:06:49.391 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:06:49.391 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.5 MB/s | 1.4 MB 00:00 2023-12-11T20:06:49.391 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:06:49.397 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:06:49.397 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:06:49.451 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:06:49.451 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:06:49.699 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:06:49.797 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-11T20:06:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-11T20:06:50.514 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-11T20:06:50.514 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:06:50.514 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:06:50.514 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:06:50.514 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:06:50.514 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:06:50.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-11T20:06:51.257 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:26 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:06:51.403 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:06:51.404 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:06:51.404 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:06:51.404 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:06:51.404 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:06:51.404 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 173 M 2023-12-11T20:06:51.404 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:06:51.404 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:06:51.404 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:06:51.405 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:06:51.405 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:06:51.405 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 173 M 2023-12-11T20:06:51.405 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 731 M 2023-12-11T20:06:51.405 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:07:07.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-radosgw-debuginfo-19.0.0-83.gc56ad3ea.el9. 11 MB/s | 173 MB 00:15 2023-12-11T20:07:07.025 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:07:07.026 INFO:teuthology.orchestra.run.smithi086.stdout:Total 11 MB/s | 173 MB 00:15 2023-12-11T20:07:07.026 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:07:07.030 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:07:07.030 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:07:08.073 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:07:08.073 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:07:11.512 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:07:11.896 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:07:15.872 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:07:16.143 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:07:16.143 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:16.144 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:07:16.144 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:07:16.144 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:16.144 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:07:16.395 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-11T20:07:16.988 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:03:55 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:07:17.147 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:07:17.147 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.4 M 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.4 M 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 6.4 M 2023-12-11T20:07:17.148 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:07:17.615 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-83 3.1 MB/s | 1.4 MB 00:00 2023-12-11T20:07:17.616 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:07:17.616 INFO:teuthology.orchestra.run.smithi133.stdout:Total 3.0 MB/s | 1.4 MB 00:00 2023-12-11T20:07:17.616 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:07:17.620 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:07:17.620 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:07:17.662 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:07:17.662 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:07:17.916 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:07:18.014 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-11T20:07:18.447 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-11T20:07:18.712 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-11T20:07:18.712 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:18.712 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:07:18.713 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:07:18.713 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:18.713 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:07:18.900 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-11T20:07:19.480 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:03:58 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:07:19.627 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:07:19.627 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:07:19.627 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:07:19.627 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 173 M 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 173 M 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 731 M 2023-12-11T20:07:19.628 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:07:25.785 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:07:25.871 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-11T20:07:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-11T20:07:30.550 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-11T20:07:30.550 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:07:30.550 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:07:30.550 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:07:30.550 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:07:30.550 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:07:30.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test-debuginfo 2023-12-11T20:07:31.319 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:06 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:07:31.462 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 676 M 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:07:31.463 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:07:31.464 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 676 M 2023-12-11T20:07:31.464 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 3.1 G 2023-12-11T20:07:31.464 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:07:31.583 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-radosgw-debuginfo-19.0.0-83.gc56ad3ea.el9. 14 MB/s | 173 MB 00:11 2023-12-11T20:07:31.584 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:07:31.585 INFO:teuthology.orchestra.run.smithi133.stdout:Total 14 MB/s | 173 MB 00:11 2023-12-11T20:07:31.585 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:07:31.589 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:07:31.589 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:07:32.634 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:07:32.635 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:07:50.655 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:07:50.788 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-11T20:07:55.177 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-11T20:07:55.459 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-11T20:07:55.460 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:55.460 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:07:55.460 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:07:55.460 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:55.460 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:07:55.661 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-test-debuginfo 2023-12-11T20:07:56.244 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:04:35 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:07:56.390 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 676 M 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:07:56.391 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:07:56.392 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 676 M 2023-12-11T20:07:56.392 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 3.1 G 2023-12-11T20:07:56.392 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:08:19.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-test-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 14 MB/s | 676 MB 00:48 2023-12-11T20:08:19.490 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:08:19.490 INFO:teuthology.orchestra.run.smithi086.stdout:Total 14 MB/s | 676 MB 00:48 2023-12-11T20:08:19.490 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:08:19.495 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:08:19.495 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:08:23.468 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:08:23.468 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:09:34.519 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:09:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:09:43.405 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:09:43.692 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:09:43.692 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:09:43.692 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:09:43.692 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:09:43.692 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:09:43.692 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:09:43.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base-debuginfo 2023-12-11T20:09:44.556 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:06:19 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:09:44.652 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:09:44.702 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:09:44.703 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-11T20:09:44.704 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:09:44.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-debuginfo 2023-12-11T20:09:45.034 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-test-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 6.2 MB/s | 676 MB 01:48 2023-12-11T20:09:45.035 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:09:45.035 INFO:teuthology.orchestra.run.smithi133.stdout:Total 6.2 MB/s | 676 MB 01:48 2023-12-11T20:09:45.036 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:09:45.041 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:09:45.041 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:09:45.364 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:06:20 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:09:45.508 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:09:45.508 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 19 M 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 19 M 2023-12-11T20:09:45.509 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 101 M 2023-12-11T20:09:45.510 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:09:46.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 13 MB/s | 19 MB 00:01 2023-12-11T20:09:46.984 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:09:46.984 INFO:teuthology.orchestra.run.smithi086.stdout:Total 13 MB/s | 19 MB 00:01 2023-12-11T20:09:46.985 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:09:46.988 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:09:46.988 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:09:47.136 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:09:47.136 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:09:49.030 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:09:49.030 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:09:49.307 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:09:49.490 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:09:50.444 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:09:50.706 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:09:50.706 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:09:50.706 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:09:50.706 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:09:50.706 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:09:50.707 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:09:50.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mds-debuginfo 2023-12-11T20:09:51.470 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:06:26 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:09:51.614 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 46 M 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 46 M 2023-12-11T20:09:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 168 M 2023-12-11T20:09:51.616 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:09:55.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mds-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 12 MB/s | 46 MB 00:03 2023-12-11T20:09:55.353 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:09:55.353 INFO:teuthology.orchestra.run.smithi086.stdout:Total 12 MB/s | 46 MB 00:03 2023-12-11T20:09:55.354 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:09:55.358 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:09:55.358 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:09:55.662 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:09:55.662 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:10:00.236 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:10:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:10:01.836 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:10:02.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mon-debuginfo 2023-12-11T20:10:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:06:37 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:10:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:10:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 111 M 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 111 M 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 506 M 2023-12-11T20:10:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:10:13.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 11 MB/s | 111 MB 00:10 2023-12-11T20:10:13.613 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:10:13.614 INFO:teuthology.orchestra.run.smithi086.stdout:Total 11 MB/s | 111 MB 00:10 2023-12-11T20:10:13.614 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:10:13.618 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:10:13.618 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:10:14.300 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:10:14.300 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:10:25.912 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:10:26.100 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:10:29.225 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:10:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:10:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:10:29.502 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:10:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:10:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:10:29.502 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:10:29.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-osd-debuginfo 2023-12-11T20:10:30.284 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:07:05 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:10:30.428 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:10:30.429 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:10:30.429 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:10:30.429 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:10:30.429 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 311 M 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 311 M 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.3 G 2023-12-11T20:10:30.430 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:10:51.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 15 MB/s | 311 MB 00:21 2023-12-11T20:10:51.637 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:10:51.638 INFO:teuthology.orchestra.run.smithi086.stdout:Total 15 MB/s | 311 MB 00:21 2023-12-11T20:10:51.638 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:10:51.642 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:10:51.642 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:10:53.495 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:10:53.495 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:10:59.969 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:11:00.215 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:07.341 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:07.611 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:07.611 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:07.611 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:11:07.611 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:07.612 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:07.612 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:11:07.895 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base-debuginfo 2023-12-11T20:11:08.482 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:07:47 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:11:08.578 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-11T20:11:08.629 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:11:08.630 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-12-11T20:11:08.631 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:11:08.699 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-debuginfo 2023-12-11T20:11:09.281 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:07:48 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:11:09.426 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:11:09.427 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:09.427 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:11:09.427 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:09.427 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:11:09.427 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 19 M 2023-12-11T20:11:09.428 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:09.428 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:11:09.428 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:09.428 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:11:09.428 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:09.428 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 19 M 2023-12-11T20:11:09.428 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 101 M 2023-12-11T20:11:09.428 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:11:11.610 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-mgr-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 8.8 MB/s | 19 MB 00:02 2023-12-11T20:11:11.611 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:11:11.611 INFO:teuthology.orchestra.run.smithi133.stdout:Total 8.8 MB/s | 19 MB 00:02 2023-12-11T20:11:11.611 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:11:11.615 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:11:11.616 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:11:11.764 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:11:11.765 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:11:13.939 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:11:14.032 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:14.987 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:15.254 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:15.254 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:15.254 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:11:15.254 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:15.254 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:15.254 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:11:15.431 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mds-debuginfo 2023-12-11T20:11:16.014 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:07:55 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:11:16.158 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:11:16.159 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:16.159 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:11:16.159 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:16.159 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:11:16.159 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 46 M 2023-12-11T20:11:16.160 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:16.160 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:11:16.160 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:16.160 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:11:16.160 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:16.160 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 46 M 2023-12-11T20:11:16.160 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 168 M 2023-12-11T20:11:16.160 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:11:22.953 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-mds-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 6.8 MB/s | 46 MB 00:06 2023-12-11T20:11:22.954 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:11:22.954 INFO:teuthology.orchestra.run.smithi133.stdout:Total 6.8 MB/s | 46 MB 00:06 2023-12-11T20:11:22.955 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:11:22.959 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:11:22.959 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:11:23.263 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:11:23.263 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:11:25.417 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:11:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:27.832 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:11:27.968 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:29.273 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:29.549 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:29.549 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:29.549 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:11:29.549 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:29.549 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:29.549 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:11:29.742 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mon-debuginfo 2023-12-11T20:11:30.326 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:09 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:11:30.472 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:11:30.473 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:30.473 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:11:30.473 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:30.473 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:11:30.473 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 111 M 2023-12-11T20:11:30.473 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:30.473 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:11:30.473 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:30.474 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:11:30.474 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:30.474 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 111 M 2023-12-11T20:11:30.474 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 506 M 2023-12-11T20:11:30.474 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:11:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:33.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:33.610 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:33.610 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:11:33.610 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:33.611 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:33.611 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:11:33.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse-debuginfo 2023-12-11T20:11:34.433 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:08:09 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:11:34.580 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:11:34.580 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:34.580 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:11:34.580 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:34.581 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:11:34.581 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.9 M 2023-12-11T20:11:34.581 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:34.581 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:11:34.581 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:34.581 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:11:34.581 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:34.582 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 5.9 M 2023-12-11T20:11:34.582 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 28 M 2023-12-11T20:11:34.582 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:11:37.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-fuse-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 2.4 MB/s | 5.9 MB 00:02 2023-12-11T20:11:37.036 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:11:37.037 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.4 MB/s | 5.9 MB 00:02 2023-12-11T20:11:37.037 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:11:37.041 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:11:37.041 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:11:37.111 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:11:37.111 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:11:37.835 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:11:37.933 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:38.459 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:38.734 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:38.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:38.734 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:11:38.734 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:38.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:38.735 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:11:38.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel-debuginfo 2023-12-11T20:11:39.479 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:08:14 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:11:39.624 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 108 k 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:39.625 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 108 k 2023-12-11T20:11:39.626 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 440 k 2023-12-11T20:11:39.626 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:11:39.946 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-debuginfo-19.0.0-83.gc56ad3ea.el 338 kB/s | 108 kB 00:00 2023-12-11T20:11:39.947 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:11:39.947 INFO:teuthology.orchestra.run.smithi086.stdout:Total 337 kB/s | 108 kB 00:00 2023-12-11T20:11:39.947 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:11:39.950 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:11:39.951 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:11:39.985 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:11:39.985 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:11:40.102 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:11:40.195 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:11:40.575 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:11:40.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:11:40.853 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:40.853 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:11:40.853 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:40.853 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:40.853 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:11:41.045 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-mon-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 11 MB/s | 111 MB 00:10 2023-12-11T20:11:41.046 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:11:41.047 INFO:teuthology.orchestra.run.smithi133.stdout:Total 11 MB/s | 111 MB 00:10 2023-12-11T20:11:41.047 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:11:41.051 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:11:41.051 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:11:41.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2-debuginfo 2023-12-11T20:11:41.633 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:08:16 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:11:41.740 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:11:41.740 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:11:41.777 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.3 M 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:41.778 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:11:41.779 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:41.779 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 5.3 M 2023-12-11T20:11:41.779 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 24 M 2023-12-11T20:11:41.779 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:11:42.466 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs2-debuginfo-19.0.0-83.gc56ad3ea.el9.x8 7.7 MB/s | 5.3 MB 00:00 2023-12-11T20:11:42.466 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:11:42.467 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.7 MB/s | 5.3 MB 00:00 2023-12-11T20:11:42.467 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:11:42.470 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:11:42.470 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:11:42.534 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:11:42.534 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:11:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:11:43.306 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:11:43.792 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:11:44.083 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:11:44.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:44.084 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:11:44.084 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:44.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:44.084 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:11:44.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2-debuginfo 2023-12-11T20:11:44.883 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:08:19 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:11:45.028 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:11:45.029 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:45.029 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:11:45.029 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:45.029 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:11:45.029 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 44 M 2023-12-11T20:11:45.029 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:45.030 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:11:45.030 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:45.030 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:11:45.030 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:45.030 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 44 M 2023-12-11T20:11:45.030 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 250 M 2023-12-11T20:11:45.030 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:11:47.893 INFO:teuthology.orchestra.run.smithi086.stdout:librados2-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 15 MB/s | 44 MB 00:02 2023-12-11T20:11:47.894 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:11:47.895 INFO:teuthology.orchestra.run.smithi086.stdout:Total 15 MB/s | 44 MB 00:02 2023-12-11T20:11:47.895 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:11:47.899 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:11:47.899 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:11:48.195 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:11:48.196 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:11:53.178 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:11:53.283 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:53.407 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:11:53.501 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:55.070 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:55.342 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:55.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:55.343 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:11:55.343 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:55.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:55.343 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:11:55.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1-debuginfo 2023-12-11T20:11:56.140 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:08:31 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:11:56.286 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:11:56.286 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 51 M 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 51 M 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 305 M 2023-12-11T20:11:56.287 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:11:56.699 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:56.973 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:11:56.973 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:56.974 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:11:56.974 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:11:56.974 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:56.974 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:11:57.169 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-osd-debuginfo 2023-12-11T20:11:57.761 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:36 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:11:57.908 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:11:57.908 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 311 M 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 311 M 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.3 G 2023-12-11T20:11:57.909 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:12:08.537 INFO:teuthology.orchestra.run.smithi086.stdout:librbd1-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_6 4.2 MB/s | 51 MB 00:12 2023-12-11T20:12:08.537 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:12:08.538 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.2 MB/s | 51 MB 00:12 2023-12-11T20:12:08.538 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:12:08.542 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:12:08.542 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:12:08.882 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:12:08.882 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:12:14.846 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:12:15.143 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:17.547 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:17.547 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:17.547 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:12:17.547 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:12:17.547 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:17.547 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:12:17.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs-debuginfo 2023-12-11T20:12:18.342 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:08:53 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:12:18.496 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:12:18.496 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 615 k 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:12:18.497 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:18.498 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 615 k 2023-12-11T20:12:18.498 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.1 M 2023-12-11T20:12:18.498 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:12:19.539 INFO:teuthology.orchestra.run.smithi086.stdout:python3-cephfs-debuginfo-19.0.0-83.gc56ad3ea.el 591 kB/s | 615 kB 00:01 2023-12-11T20:12:19.540 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:12:19.541 INFO:teuthology.orchestra.run.smithi086.stdout:Total 590 kB/s | 615 kB 00:01 2023-12-11T20:12:19.541 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:12:19.544 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:12:19.544 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:12:19.580 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:12:19.580 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:12:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:12:19.837 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:12:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:12:20.461 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:12:20.461 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:20.461 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:12:20.462 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:12:20.462 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:20.462 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:12:20.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados-debuginfo 2023-12-11T20:12:21.237 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:08:56 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:12:21.384 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 770 k 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:21.385 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 770 k 2023-12-11T20:12:21.386 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.7 M 2023-12-11T20:12:21.386 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:12:22.250 INFO:teuthology.orchestra.run.smithi086.stdout:python3-rados-debuginfo-19.0.0-83.gc56ad3ea.el9 890 kB/s | 770 kB 00:00 2023-12-11T20:12:22.250 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:12:22.250 INFO:teuthology.orchestra.run.smithi086.stdout:Total 889 kB/s | 770 kB 00:00 2023-12-11T20:12:22.251 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:12:22.254 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:12:22.255 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:12:22.293 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:12:22.293 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:12:22.476 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:12:22.574 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-11T20:12:22.938 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-11T20:12:23.205 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-11T20:12:23.205 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:23.205 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:12:23.206 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:12:23.206 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:23.206 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:12:23.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd-debuginfo 2023-12-11T20:12:23.987 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:08:58 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:12:24.131 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:12:24.131 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:24.131 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:12:24.131 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:24.131 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:12:24.131 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.1 M 2023-12-11T20:12:24.131 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:24.132 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:12:24.132 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:24.132 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:12:24.132 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:24.133 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.1 M 2023-12-11T20:12:24.133 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 4.3 M 2023-12-11T20:12:24.133 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:12:24.861 INFO:teuthology.orchestra.run.smithi086.stdout:python3-rbd-debuginfo-19.0.0-83.gc56ad3ea.el9.x 1.5 MB/s | 1.1 MB 00:00 2023-12-11T20:12:24.862 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:12:24.862 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.5 MB/s | 1.1 MB 00:00 2023-12-11T20:12:24.862 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:12:24.866 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:12:24.866 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:12:24.907 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:12:24.908 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:12:25.123 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:12:25.235 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:12:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:12:25.878 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:12:25.878 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:25.878 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:12:25.878 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:12:25.878 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:25.878 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:12:26.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw-debuginfo 2023-12-11T20:12:26.671 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:09:01 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:12:26.816 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:12:26.818 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:26.818 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:12:26.818 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:26.818 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:12:26.818 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 337 k 2023-12-11T20:12:26.818 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:26.818 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:12:26.819 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:26.819 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:12:26.819 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:26.819 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 337 k 2023-12-11T20:12:26.819 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.0 M 2023-12-11T20:12:26.819 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:12:27.464 INFO:teuthology.orchestra.run.smithi086.stdout:python3-rgw-debuginfo-19.0.0-83.gc56ad3ea.el9.x 522 kB/s | 337 kB 00:00 2023-12-11T20:12:27.464 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:12:27.465 INFO:teuthology.orchestra.run.smithi086.stdout:Total 521 kB/s | 337 kB 00:00 2023-12-11T20:12:27.465 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:12:27.468 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:12:27.468 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:12:27.503 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:12:27.504 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:12:27.640 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:12:27.738 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:12:28.101 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:12:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:12:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:28.377 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:12:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:12:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:28.378 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:12:28.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse-debuginfo 2023-12-11T20:12:29.166 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:09:04 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:12:29.326 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:12:29.326 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:29.326 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:12:29.326 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:29.326 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:12:29.326 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 508 k 2023-12-11T20:12:29.326 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:29.327 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:12:29.327 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:29.327 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:12:29.327 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:29.327 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 508 k 2023-12-11T20:12:29.327 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.5 M 2023-12-11T20:12:29.327 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:12:30.167 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 603 kB/s | 508 kB 00:00 2023-12-11T20:12:30.167 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:12:30.168 INFO:teuthology.orchestra.run.smithi086.stdout:Total 602 kB/s | 508 kB 00:00 2023-12-11T20:12:30.168 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:12:30.172 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:12:30.172 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:12:30.210 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:12:30.210 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:12:30.371 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:12:30.466 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:31.154 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:31.154 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:31.154 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:12:31.154 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:12:31.154 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:31.154 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:12:31.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror-debuginfo 2023-12-11T20:12:31.951 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:09:06 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:12:32.096 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:12:32.097 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:32.097 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:12:32.097 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:32.097 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:12:32.097 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 51 M 2023-12-11T20:12:32.097 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:32.097 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:12:32.097 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:32.098 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:12:32.098 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:32.098 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 51 M 2023-12-11T20:12:32.098 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 296 M 2023-12-11T20:12:32.098 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:12:40.369 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-debuginfo-19.0.0-83.gc56ad3ea.el9.x8 6.1 MB/s | 51 MB 00:08 2023-12-11T20:12:40.370 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:12:40.370 INFO:teuthology.orchestra.run.smithi086.stdout:Total 6.1 MB/s | 51 MB 00:08 2023-12-11T20:12:40.370 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:12:40.374 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:12:40.374 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:12:40.708 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:12:40.708 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:12:46.552 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:12:46.649 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:12:48.649 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:12:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:12:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:48.917 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:12:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:12:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:48.918 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:12:49.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd-debuginfo 2023-12-11T20:12:49.705 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:09:24 ago on Mon 11 Dec 2023 08:03:25 PM UTC. 2023-12-11T20:12:49.853 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.2 M 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-11T20:12:49.854 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-11T20:12:49.855 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:49.855 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.2 M 2023-12-11T20:12:49.855 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 5.4 M 2023-12-11T20:12:49.855 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-11T20:12:50.821 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_6 1.3 MB/s | 1.2 MB 00:00 2023-12-11T20:12:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:12:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.3 MB/s | 1.2 MB 00:00 2023-12-11T20:12:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-11T20:12:50.826 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-11T20:12:50.827 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-11T20:12:50.870 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-11T20:12:50.870 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-11T20:12:51.118 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-11T20:12:51.210 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:51.592 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:12:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:51.866 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-11T20:12:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:12:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:12:51.866 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-11T20:12:52.054 DEBUG:teuthology.parallel:result is None 2023-12-11T20:13:08.996 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-osd-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 4.4 MB/s | 311 MB 01:11 2023-12-11T20:13:08.996 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:13:08.996 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.4 MB/s | 311 MB 01:11 2023-12-11T20:13:08.997 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:13:09.000 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:13:09.000 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:13:10.845 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:13:10.845 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:13:42.750 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:13:42.839 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:13:50.672 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:13:50.948 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:13:50.948 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:50.948 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:13:50.948 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:13:50.948 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:50.948 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:13:51.180 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-fuse-debuginfo 2023-12-11T20:13:51.768 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:10:30 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:13:51.912 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:13:51.913 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.9 M 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:13:51.914 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:51.915 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 5.9 M 2023-12-11T20:13:51.915 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 28 M 2023-12-11T20:13:51.915 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:13:52.584 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-fuse-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 8.8 MB/s | 5.9 MB 00:00 2023-12-11T20:13:52.584 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:13:52.585 INFO:teuthology.orchestra.run.smithi133.stdout:Total 8.8 MB/s | 5.9 MB 00:00 2023-12-11T20:13:52.585 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:13:52.588 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:13:52.588 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:13:52.655 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:13:52.656 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:13:53.386 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:13:53.473 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:13:54.008 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:13:54.273 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:13:54.273 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:54.273 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:13:54.273 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:13:54.273 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:54.273 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:13:54.444 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados-devel-debuginfo 2023-12-11T20:13:55.025 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:10:34 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:13:55.171 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:13:55.172 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:55.172 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:13:55.172 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:55.172 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 108 k 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 108 k 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 440 k 2023-12-11T20:13:55.173 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:13:55.487 INFO:teuthology.orchestra.run.smithi133.stdout:librados-devel-debuginfo-19.0.0-83.gc56ad3ea.el 346 kB/s | 108 kB 00:00 2023-12-11T20:13:55.487 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:13:55.488 INFO:teuthology.orchestra.run.smithi133.stdout:Total 344 kB/s | 108 kB 00:00 2023-12-11T20:13:55.488 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:13:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:13:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:13:55.524 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:13:55.525 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:13:55.636 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:13:55.735 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:13:56.074 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:13:56.343 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:13:56.343 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:56.343 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:13:56.343 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:13:56.343 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:56.343 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:13:56.517 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs2-debuginfo 2023-12-11T20:13:57.103 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:10:36 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:13:57.248 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.3 M 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 5.3 M 2023-12-11T20:13:57.249 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 24 M 2023-12-11T20:13:57.250 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:00.531 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs2-debuginfo-19.0.0-83.gc56ad3ea.el9.x8 1.6 MB/s | 5.3 MB 00:03 2023-12-11T20:14:00.532 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:00.532 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.6 MB/s | 5.3 MB 00:03 2023-12-11T20:14:00.532 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:00.536 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:00.536 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:00.604 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:00.604 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:14:01.265 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:14:01.362 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:14:01.858 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:14:02.131 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:14:02.131 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:02.131 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:14:02.131 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:14:02.131 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:02.131 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:14:02.337 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados2-debuginfo 2023-12-11T20:14:02.925 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:10:41 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:14:03.072 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:14:03.072 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:03.072 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:14:03.073 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:03.073 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:14:03.073 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 44 M 2023-12-11T20:14:03.073 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:03.073 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:14:03.073 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:03.073 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:14:03.073 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:03.074 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 44 M 2023-12-11T20:14:03.074 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 250 M 2023-12-11T20:14:03.074 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:10.586 INFO:teuthology.orchestra.run.smithi133.stdout:librados2-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 5.8 MB/s | 44 MB 00:07 2023-12-11T20:14:10.586 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:10.586 INFO:teuthology.orchestra.run.smithi133.stdout:Total 5.8 MB/s | 44 MB 00:07 2023-12-11T20:14:10.587 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:10.588 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:10.588 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:10.887 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:10.888 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:14:15.877 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:14:15.972 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:17.747 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:18.028 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:18.028 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:18.029 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:14:18.029 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:14:18.029 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:18.029 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:14:18.263 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librbd1-debuginfo 2023-12-11T20:14:18.847 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:10:57 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:14:18.994 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:14:19.010 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:19.010 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:14:19.010 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 51 M 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 51 M 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 305 M 2023-12-11T20:14:19.011 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:34.741 INFO:teuthology.orchestra.run.smithi133.stdout:librbd1-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_6 3.2 MB/s | 51 MB 00:15 2023-12-11T20:14:34.743 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:34.743 INFO:teuthology.orchestra.run.smithi133.stdout:Total 3.2 MB/s | 51 MB 00:15 2023-12-11T20:14:34.743 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:34.746 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:34.746 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:35.092 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:35.092 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:14:41.038 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:14:41.138 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:43.235 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:43.519 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:43.520 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:43.520 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:14:43.520 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:14:43.520 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:43.520 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:14:43.730 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-cephfs-debuginfo 2023-12-11T20:14:44.312 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:11:23 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:14:44.458 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 615 k 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 615 k 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.1 M 2023-12-11T20:14:44.459 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:45.143 INFO:teuthology.orchestra.run.smithi133.stdout:python3-cephfs-debuginfo-19.0.0-83.gc56ad3ea.el 900 kB/s | 615 kB 00:00 2023-12-11T20:14:45.144 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:45.144 INFO:teuthology.orchestra.run.smithi133.stdout:Total 898 kB/s | 615 kB 00:00 2023-12-11T20:14:45.144 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:45.147 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:45.147 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:45.185 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:45.186 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:14:45.353 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:14:45.450 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:14:45.874 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:14:46.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-11T20:14:46.147 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:46.148 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:14:46.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:14:46.148 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:46.148 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:14:46.359 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rados-debuginfo 2023-12-11T20:14:46.941 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:11:25 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:14:47.085 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:14:47.086 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:47.086 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:14:47.086 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:47.086 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:14:47.086 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 770 k 2023-12-11T20:14:47.087 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:47.087 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:14:47.087 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:47.087 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:14:47.087 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:47.087 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 770 k 2023-12-11T20:14:47.087 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.7 M 2023-12-11T20:14:47.087 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:47.371 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rados-debuginfo-19.0.0-83.gc56ad3ea.el9 2.6 MB/s | 770 kB 00:00 2023-12-11T20:14:47.372 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:47.372 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.6 MB/s | 770 kB 00:00 2023-12-11T20:14:47.372 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:47.375 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:47.375 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:47.412 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:47.412 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:14:47.594 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:14:47.703 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-11T20:14:48.082 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-11T20:14:48.354 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-11T20:14:48.355 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:48.355 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:14:48.355 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:14:48.355 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:48.355 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:14:48.568 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rbd-debuginfo 2023-12-11T20:14:49.160 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:11:28 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:14:49.307 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.1 M 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:49.308 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:14:49.309 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:49.309 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.1 M 2023-12-11T20:14:49.309 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 4.3 M 2023-12-11T20:14:49.309 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:49.765 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rbd-debuginfo-19.0.0-83.gc56ad3ea.el9.x 2.5 MB/s | 1.1 MB 00:00 2023-12-11T20:14:49.766 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:49.766 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.5 MB/s | 1.1 MB 00:00 2023-12-11T20:14:49.766 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:49.770 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:49.770 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:49.810 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:49.810 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:14:50.024 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:14:50.122 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:14:50.506 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:14:50.794 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:14:50.794 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:50.794 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:14:50.794 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:14:50.794 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:50.795 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:14:51.009 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rgw-debuginfo 2023-12-11T20:14:51.595 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:11:30 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:14:51.745 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:14:51.745 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:51.745 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:14:51.745 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:51.745 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:14:51.746 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 337 k 2023-12-11T20:14:51.746 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:51.746 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:14:51.746 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:51.746 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:14:51.746 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:51.746 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 337 k 2023-12-11T20:14:51.746 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.0 M 2023-12-11T20:14:51.747 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:52.002 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rgw-debuginfo-19.0.0-83.gc56ad3ea.el9.x 1.3 MB/s | 337 kB 00:00 2023-12-11T20:14:52.003 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:52.003 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.3 MB/s | 337 kB 00:00 2023-12-11T20:14:52.003 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:52.006 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:52.006 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:52.041 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:52.042 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:14:52.176 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:14:52.291 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:14:52.682 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:14:52.964 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-11T20:14:52.965 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:52.965 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:14:52.965 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:14:52.965 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:52.965 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:14:53.158 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-fuse-debuginfo 2023-12-11T20:14:53.744 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:11:32 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:14:53.889 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:14:53.890 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:53.890 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:14:53.890 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:53.890 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:14:53.890 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 508 k 2023-12-11T20:14:53.890 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:53.890 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:14:53.890 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:53.891 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:14:53.891 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:53.891 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 508 k 2023-12-11T20:14:53.891 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.5 M 2023-12-11T20:14:53.891 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:54.264 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 1.3 MB/s | 508 kB 00:00 2023-12-11T20:14:54.264 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:54.264 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.3 MB/s | 508 kB 00:00 2023-12-11T20:14:54.265 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:54.268 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:54.268 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:54.304 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:54.305 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:14:54.465 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:14:54.561 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:54.962 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:55.248 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:14:55.248 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:55.248 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:14:55.248 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:14:55.248 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:55.248 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:14:55.455 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-mirror-debuginfo 2023-12-11T20:14:56.043 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:11:35 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:14:56.190 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:14:56.190 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:56.190 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:14:56.190 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 51 M 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 51 M 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 296 M 2023-12-11T20:14:56.191 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:14:59.100 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-mirror-debuginfo-19.0.0-83.gc56ad3ea.el9.x8 17 MB/s | 51 MB 00:02 2023-12-11T20:14:59.101 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:14:59.101 INFO:teuthology.orchestra.run.smithi133.stdout:Total 17 MB/s | 51 MB 00:02 2023-12-11T20:14:59.101 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:14:59.105 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:14:59.105 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:14:59.435 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:14:59.435 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:15:05.258 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:15:05.346 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:15:07.347 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:15:07.629 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-11T20:15:07.629 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:07.629 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:15:07.629 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:15:07.629 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:07.629 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:15:07.832 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-nbd-debuginfo 2023-12-11T20:15:08.419 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:11:47 ago on Mon 11 Dec 2023 08:03:21 PM UTC. 2023-12-11T20:15:08.567 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-12-11T20:15:08.567 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.2 M 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-12-11T20:15:08.568 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:08.569 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.2 M 2023-12-11T20:15:08.569 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 5.4 M 2023-12-11T20:15:08.569 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-12-11T20:15:09.434 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-nbd-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_6 1.4 MB/s | 1.2 MB 00:00 2023-12-11T20:15:09.435 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-12-11T20:15:09.435 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.4 MB/s | 1.2 MB 00:00 2023-12-11T20:15:09.436 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-12-11T20:15:09.439 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-12-11T20:15:09.439 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-12-11T20:15:09.482 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-12-11T20:15:09.483 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-12-11T20:15:09.723 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-12-11T20:15:09.824 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:15:10.232 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:15:10.498 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-11T20:15:10.498 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:10.498 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-12-11T20:15:10.498 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-11T20:15:10.498 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:10.498 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-12-11T20:15:10.719 DEBUG:teuthology.parallel:result is None 2023-12-11T20:15:10.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-11T20:15:10.862 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-11T20:15:10.887 INFO:teuthology.orchestra.run.smithi086.stdout:19.0.0-83.gc56ad3ea.el9 2023-12-11T20:15:10.888 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-83.gc56ad3ea.el9 2023-12-11T20:15:10.888 INFO:teuthology.task.install:The correct ceph version 19.0.0-83.gc56ad3ea is installed. 2023-12-11T20:15:10.889 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-11T20:15:11.027 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-11T20:15:11.048 INFO:teuthology.orchestra.run.smithi133.stdout:19.0.0-83.gc56ad3ea.el9 2023-12-11T20:15:11.048 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-83.gc56ad3ea.el9 2023-12-11T20:15:11.048 INFO:teuthology.task.install:The correct ceph version 19.0.0-83.gc56ad3ea is installed. 2023-12-11T20:15:11.050 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-11T20:15:11.050 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:11.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-11T20:15:11.082 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:11.083 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-11T20:15:11.115 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-11T20:15:11.116 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:11.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2023-12-11T20:15:11.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-11T20:15:11.224 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:11.224 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2023-12-11T20:15:11.251 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-11T20:15:11.318 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-11T20:15:11.318 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:11.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-11T20:15:11.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-11T20:15:11.415 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:11.415 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-11T20:15:11.442 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-11T20:15:11.509 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-11T20:15:11.510 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:11.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2023-12-11T20:15:11.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-11T20:15:11.605 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:11.605 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/stdin-killer 2023-12-11T20:15:11.632 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-11T20:15:11.700 INFO:teuthology.run_tasks:Running task ceph... 2023-12-11T20:15:11.792 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-11T20:15:11.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2023-12-11T20:15:11.795 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /var/log/ceph 2023-12-11T20:15:11.822 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-11T20:15:11.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-11T20:15:11.863 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-11T20:15:11.889 INFO:tasks.ceph:Creating extra log directories... 2023-12-11T20:15:11.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-11T20:15:11.931 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-11T20:15:11.968 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-11T20:15:11.968 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-11T20:15:11.968 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265', 'branch': 'wip-yuri3-testing-2023-12-07-0727', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7487265', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-12-07-0727'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2023-12-07-0727', 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9380, 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri3-testing-2023-12-07-0727', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'targets': {'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMOJn5q+AoDi076ehgEb2UB7o6YSW8kCF1AYQaNGSo6OsPXHsWuu+qyowrTGgX9c9Y5SWcZZUcD7KSBtoPIZFXI=', 'smithi133.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvBdaZakCDp41o3yJ8GUZkZbVVpPi2BSMpZMLhkk3GfTViBdqH1gJ1b4iArrB2Z2NsODYFNDvs6n8UV3I9Ceps='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-11_16:41:51', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-11T20:15:11.969 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-11T20:15:11.998 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-11T20:15:12.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2023-12-11T20:15:12.053 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2023-12-11T20:15:12.080 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:12.080 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2023-12-11T20:15:12.133 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-11T20:15:12.133 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2023-12-11T20:15:12.188 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-11T20:15:12.189 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-11T20:15:12.189 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-11T20:15:12.189 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-11T20:15:12.189 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-12-11T20:15:12.189 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-12-11 20:05:12.607562130 +0000 2023-12-11T20:15:12.189 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-12-11 20:02:25.767778323 +0000 2023-12-11T20:15:12.189 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-12-11 20:02:25.767778323 +0000 2023-12-11T20:15:12.189 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2023-12-11 20:02:25.767778323 +0000 2023-12-11T20:15:12.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-11T20:15:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-12-11T20:15:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-12-11T20:15:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000179261 s, 2.9 MB/s 2023-12-11T20:15:12.255 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-11T20:15:12.311 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-12-11 20:05:12.607562130 +0000 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-12-11 20:02:26.316766544 +0000 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-12-11 20:02:26.316766544 +0000 2023-12-11T20:15:12.366 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2023-12-11 20:02:26.316766544 +0000 2023-12-11T20:15:12.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-11T20:15:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-12-11T20:15:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-12-11T20:15:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000184256 s, 2.8 MB/s 2023-12-11T20:15:12.432 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-11T20:15:12.488 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2023-12-11T20:15:12.544 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-11T20:15:12.545 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-11T20:15:12.545 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-11T20:15:12.545 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-11T20:15:12.545 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-12-11T20:15:12.545 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-12-11 20:05:12.608562109 +0000 2023-12-11T20:15:12.545 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-12-11 20:02:26.817755795 +0000 2023-12-11T20:15:12.546 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-12-11 20:02:26.817755795 +0000 2023-12-11T20:15:12.546 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2023-12-11 20:02:26.817755795 +0000 2023-12-11T20:15:12.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-11T20:15:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-12-11T20:15:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-12-11T20:15:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.00023894 s, 2.1 MB/s 2023-12-11T20:15:12.612 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-11T20:15:12.667 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2023-12-11T20:15:12.722 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-11T20:15:12.723 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-11T20:15:12.723 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-11T20:15:12.723 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-11T20:15:12.723 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-12-11T20:15:12.723 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-12-11 20:05:12.608562109 +0000 2023-12-11T20:15:12.723 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-12-11 20:02:27.311745197 +0000 2023-12-11T20:15:12.723 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-12-11 20:02:27.311745197 +0000 2023-12-11T20:15:12.723 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2023-12-11 20:02:27.311745197 +0000 2023-12-11T20:15:12.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-11T20:15:12.788 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-12-11T20:15:12.788 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-12-11T20:15:12.788 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000187583 s, 2.7 MB/s 2023-12-11T20:15:12.789 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-11T20:15:12.845 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:12.845 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:12.845 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2023-12-11T20:15:12.862 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-11T20:15:12.862 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 2023-12-11T20:15:12.918 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-11T20:15:12.919 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-11T20:15:12.919 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-11T20:15:12.919 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-11T20:15:12.919 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2023-12-11T20:15:12.919 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-12-11 20:05:19.160323868 +0000 2023-12-11T20:15:12.919 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-12-11 20:02:25.760385825 +0000 2023-12-11T20:15:12.919 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-12-11 20:02:25.760385825 +0000 2023-12-11T20:15:12.919 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2023-12-11 20:02:25.760385825 +0000 2023-12-11T20:15:12.919 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-11T20:15:12.985 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-12-11T20:15:12.985 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-12-11T20:15:12.985 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000232917 s, 2.2 MB/s 2023-12-11T20:15:12.987 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-11T20:15:13.044 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-12-11 20:05:19.160323868 +0000 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-12-11 20:02:26.307376130 +0000 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-12-11 20:02:26.307376130 +0000 2023-12-11T20:15:13.100 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2023-12-11 20:02:26.307376130 +0000 2023-12-11T20:15:13.100 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-11T20:15:13.166 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-12-11T20:15:13.166 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-12-11T20:15:13.166 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000206381 s, 2.5 MB/s 2023-12-11T20:15:13.168 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-11T20:15:13.224 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 2023-12-11T20:15:13.279 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-11T20:15:13.279 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-11T20:15:13.279 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-12-11T20:15:13.279 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-11T20:15:13.279 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2023-12-11T20:15:13.279 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-12-11 20:05:19.160323868 +0000 2023-12-11T20:15:13.279 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-12-11 20:02:26.816367110 +0000 2023-12-11T20:15:13.280 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-12-11 20:02:26.816367110 +0000 2023-12-11T20:15:13.280 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2023-12-11 20:02:26.816367110 +0000 2023-12-11T20:15:13.280 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-11T20:15:13.344 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-12-11T20:15:13.345 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-12-11T20:15:13.345 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000176762 s, 2.9 MB/s 2023-12-11T20:15:13.346 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-11T20:15:13.402 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-12-11 20:05:19.160323868 +0000 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-12-11 20:02:27.324358106 +0000 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-12-11 20:02:27.324358106 +0000 2023-12-11T20:15:13.457 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2023-12-11 20:02:27.324358106 +0000 2023-12-11T20:15:13.458 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-11T20:15:13.522 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-12-11T20:15:13.522 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-12-11T20:15:13.523 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000241571 s, 2.1 MB/s 2023-12-11T20:15:13.524 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-11T20:15:13.580 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:13.580 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-11T20:15:13.580 INFO:tasks.ceph:Generating config... 2023-12-11T20:15:13.581 INFO:tasks.ceph:[client] debug monc = 20 2023-12-11T20:15:13.581 INFO:tasks.ceph:[client] debug ms = 1 2023-12-11T20:15:13.581 INFO:tasks.ceph:[client] debug objecter = 20 2023-12-11T20:15:13.581 INFO:tasks.ceph:[client] debug rados = 20 2023-12-11T20:15:13.581 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-11T20:15:13.581 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-12-11T20:15:13.582 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-11T20:15:13.582 INFO:tasks.ceph:[global] ms type = async 2023-12-11T20:15:13.582 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-11T20:15:13.582 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-12-11T20:15:13.583 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-11T20:15:13.583 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd class default list = * 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd class load list = * 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-11T20:15:13.584 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-11T20:15:13.584 INFO:tasks.ceph:Setting up mon.a... 2023-12-11T20:15:13.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-11T20:15:13.640 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.keyring 2023-12-11T20:15:13.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-11T20:15:13.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-11T20:15:13.753 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.86'), ('mon.c', '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]'), ('mon.b', '172.21.15.133')] 2023-12-11T20:15:13.753 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790],172.21.15.133', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-11T20:15:13.755 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:13.755 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-11T20:15:13.808 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.86 --addv c '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]' --add b 172.21.15.133 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:13.892 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:13.892 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: generated fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:13.892 INFO:teuthology.orchestra.run.smithi086.stdout:setting min_mon_release = pacific 2023-12-11T20:15:13.892 INFO:teuthology.orchestra.run.smithi086.stdout:epoch 0 2023-12-11T20:15:13.892 INFO:teuthology.orchestra.run.smithi086.stdout:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:13.892 INFO:teuthology.orchestra.run.smithi086.stdout:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:13.892 INFO:teuthology.orchestra.run.smithi086.stdout:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:13.892 INFO:teuthology.orchestra.run.smithi086.stdout:min_mon_release 16 (pacific) 2023-12-11T20:15:13.893 INFO:teuthology.orchestra.run.smithi086.stdout:election_strategy: 1 2023-12-11T20:15:13.893 INFO:teuthology.orchestra.run.smithi086.stdout:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:15:13.893 INFO:teuthology.orchestra.run.smithi086.stdout:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:15:13.893 INFO:teuthology.orchestra.run.smithi086.stdout:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:15:13.893 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-11T20:15:13.894 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-11T20:15:13.908 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bfc699ed-5134-41ce-a2e4-de833b93556d... 2023-12-11T20:15:13.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-11T20:15:13.951 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2023-12-11T20:15:13.999 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2023-12-11T20:15:14.000 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.001 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790],172.21.15.133" 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: mon client directed command retry = 5 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: mon election default strategy = 1 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject socket failures = 5000 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: ms type = async 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: osd heartbeat grace = 80 2023-12-11T20:15:14.002 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2023-12-11T20:15:14.003 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug trim objects = true 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.004 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: bdev async discard = True 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: bdev enable discard = True 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore allocator = bitmap 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore zero block detection = True 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 1 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 1 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 10 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2023-12-11T20:15:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd class default list = * 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd class load list = * 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd client watch timeout = 120 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify cached snaps = True 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify missing on start = True 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd fast shutdown = False 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock override recovery settings = True 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue = debug_random 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue cut off = debug_random 2023-12-11T20:15:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: osd shutdown pgref assert = True 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-12-11T20:15:14.007 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-11T20:15:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon min osdmap epochs = 50 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd crush smoke test = False 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune interval = 2 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune min = 15 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune txsize = 2 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon scrub interval = 300 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on pool no app = False 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: paxos service trim min = 10 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2023-12-11T20:15:14.009 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout: debug objecter = 20 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout: debug rados = 20 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2023-12-11T20:15:14.010 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2023-12-11T20:15:14.011 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2023-12-11T20:15:14.011 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2023-12-11T20:15:14.011 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-11T20:15:14.011 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2023-12-11T20:15:14.011 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.011 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2023-12-11T20:15:14.011 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.011 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2023-12-11T20:15:14.012 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: mon_allow_pool_size_one = true 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.013 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: mon health detail to clog = false 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = "172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790],172.21.15.133" 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: mon client directed command retry = 5 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: mon election default strategy = 1 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: ms inject socket failures = 5000 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: ms type = async 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: osd heartbeat grace = 80 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.014 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: osd mclock profile = high_recovery_ops 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2023-12-11T20:15:14.015 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug trim objects = true 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2023-12-11T20:15:14.016 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: bdev async discard = True 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: bdev enable discard = True 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore allocator = bitmap 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore block size = 96636764160 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore fsck on mount = True 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore zero block detection = True 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: debug bluefs = 1 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: debug bluestore = 1 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: debug monc = 20 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2023-12-11T20:15:14.017 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 20 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: debug rocksdb = 10 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd backfillfull_ratio = 0.85 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd full ratio = 0.9 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd nearfull ratio = 0.8 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd class default list = * 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd class load list = * 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd client watch timeout = 120 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify cached snaps = True 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify missing on start = True 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd failsafe full ratio = 0.95 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd fast shutdown = False 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd mclock override recovery settings = True 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd objectstore = bluestore 2023-12-11T20:15:14.018 INFO:teuthology.orchestra.run.smithi133.stdout: osd op queue = debug_random 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: osd op queue cut off = debug_random 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: osd shutdown pgref assert = True 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2023-12-11T20:15:14.019 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: auth mon ticket ttl = 660 # 11m 2023-12-11T20:15:14.020 INFO:teuthology.orchestra.run.smithi133.stdout: auth service ticket ttl = 240 # 4m 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: # don't complain about insecure global_id in the test suite 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: # 1m isn't quite enough 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon_down_mkfs_grace = 2m 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_filestore_osds = false 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon min osdmap epochs = 50 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd crush smoke test = False 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune interval = 2 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune min = 15 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune txsize = 2 2023-12-11T20:15:14.021 INFO:teuthology.orchestra.run.smithi133.stdout: mon scrub interval = 300 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on pool no app = False 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: paxos service trim min = 10 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: debug monc = 20 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: debug objecter = 20 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout: debug rados = 20 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2023-12-11T20:15:14.022 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.c] 2023-12-11T20:15:14.023 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.b] 2023-12-11T20:15:14.024 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-11T20:15:14.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-11T20:15:14.119 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-11T20:15:14.120 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:14.120 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-11T20:15:14.134 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:14.134 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-11T20:15:14.188 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:15:14.189 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:14.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-11T20:15:14.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-11T20:15:14.268 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:14.268 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:14.322 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:15:14.323 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:14.323 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-11T20:15:14.323 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-11T20:15:14.363 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:14.363 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:14.418 INFO:tasks.ceph:Setting up mon nodes... 2023-12-11T20:15:14.418 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-11T20:15:14.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-11T20:15:14.473 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-11T20:15:14.475 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-11T20:15:14.545 INFO:teuthology.orchestra.run.smithi133.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-11T20:15:14.547 INFO:tasks.ceph:Setting up mds nodes... 2023-12-11T20:15:14.547 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-11T20:15:14.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-11T20:15:14.589 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-11T20:15:14.604 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-11T20:15:14.646 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-11T20:15:14.661 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-11T20:15:14.661 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-11T20:15:14.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-11T20:15:14.688 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:14.688 INFO:tasks.ceph:role: osd.0 2023-12-11T20:15:14.688 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:15:14.689 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-11T20:15:14.762 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-11T20:15:14.763 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-11T20:15:15.279 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-12-11T20:15:15.281 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-12-11T20:15:15.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-11T20:15:15.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-11T20:15:15.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-11T20:15:15.522 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:15.522 INFO:tasks.ceph:role: osd.1 2023-12-11T20:15:15.522 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:15:15.522 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-11T20:15:15.597 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-11T20:15:15.597 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-11T20:15:15.597 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-11T20:15:15.597 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-11T20:15:15.598 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-11T20:15:15.598 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-12-11T20:15:15.598 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-11T20:15:15.598 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-11T20:15:15.598 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-11T20:15:15.598 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-11T20:15:16.051 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-12-11T20:15:16.054 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-12-11T20:15:16.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-11T20:15:16.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-11T20:15:16.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-11T20:15:16.256 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:16.256 INFO:tasks.ceph:role: osd.2 2023-12-11T20:15:16.256 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:15:16.256 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-11T20:15:16.331 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-11T20:15:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-11T20:15:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-11T20:15:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-11T20:15:16.332 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-11T20:15:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-12-11T20:15:16.332 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-11T20:15:16.332 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-11T20:15:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-11T20:15:16.332 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-11T20:15:16.823 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-12-11T20:15:16.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-12-11T20:15:16.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-11T20:15:16.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-11T20:15:17.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-11T20:15:17.088 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:17.088 INFO:tasks.ceph:role: osd.3 2023-12-11T20:15:17.088 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:15:17.088 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-11T20:15:17.164 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-11T20:15:17.164 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-11T20:15:17.165 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-11T20:15:17.165 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-11T20:15:17.165 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-11T20:15:17.165 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-12-11T20:15:17.165 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-11T20:15:17.165 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-11T20:15:17.165 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-11T20:15:17.165 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-11T20:15:17.676 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-12-11T20:15:17.679 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-12-11T20:15:17.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-11T20:15:17.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-11T20:15:17.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:17.960+0000 7fdae5a1a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-11T20:15:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:17.961+0000 7fdae5a1a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-11T20:15:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:17.961+0000 7fdae5a1a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-11T20:15:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:17.961+0000 7fdae5a1a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-11T20:15:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:17.961+0000 7fdae5a1a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-11T20:15:17.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:17.961+0000 7fdae5a1a500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-11T20:15:20.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-11T20:15:20.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:20.943 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:20.943+0000 7fcc75c0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-11T20:15:20.943 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:20.943+0000 7fcc75c0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-11T20:15:20.943 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:20.943+0000 7fcc75c0e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-11T20:15:20.943 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:20.943+0000 7fcc75c0e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-11T20:15:20.943 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:20.943+0000 7fcc75c0e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-11T20:15:20.943 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:20.944+0000 7fcc75c0e500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-11T20:15:23.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-11T20:15:23.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:23.909 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:23.911+0000 7f5d98e10500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-11T20:15:23.910 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:23.911+0000 7f5d98e10500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-11T20:15:23.910 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:23.911+0000 7f5d98e10500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-11T20:15:23.910 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:23.911+0000 7f5d98e10500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-11T20:15:23.910 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:23.911+0000 7f5d98e10500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-11T20:15:23.910 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:23.912+0000 7f5d98e10500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-11T20:15:26.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-11T20:15:26.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:26.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:26.876+0000 7f8407416500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-11T20:15:26.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:26.876+0000 7f8407416500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-11T20:15:26.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:26.876+0000 7f8407416500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-11T20:15:26.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:26.876+0000 7f8407416500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-11T20:15:26.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:26.876+0000 7f8407416500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-11T20:15:26.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:26.877+0000 7f8407416500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-11T20:15:29.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-11T20:15:29.750 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-11T20:15:29.782 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:29.782 INFO:tasks.ceph:role: osd.4 2023-12-11T20:15:29.782 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:15:29.782 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-11T20:15:29.861 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-11T20:15:29.862 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-11T20:15:30.358 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2023-12-11T20:15:30.360 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2023-12-11T20:15:30.360 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-11T20:15:30.512 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-11T20:15:30.549 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-11T20:15:30.618 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:30.618 INFO:tasks.ceph:role: osd.5 2023-12-11T20:15:30.619 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:15:30.619 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-11T20:15:30.693 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-11T20:15:30.693 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-11T20:15:30.693 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-11T20:15:30.693 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-11T20:15:30.693 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-11T20:15:30.693 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2023-12-11T20:15:30.694 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-11T20:15:30.694 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-11T20:15:30.694 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-11T20:15:30.694 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-11T20:15:31.135 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2023-12-11T20:15:31.137 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2023-12-11T20:15:31.138 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-11T20:15:31.228 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-11T20:15:31.260 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-11T20:15:31.329 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:31.329 INFO:tasks.ceph:role: osd.6 2023-12-11T20:15:31.329 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:15:31.329 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-11T20:15:31.404 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-11T20:15:31.405 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-11T20:15:31.886 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2023-12-11T20:15:31.887 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2023-12-11T20:15:31.888 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-11T20:15:32.015 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-11T20:15:32.050 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-11T20:15:32.119 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-11T20:15:32.119 INFO:tasks.ceph:role: osd.7 2023-12-11T20:15:32.119 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:15:32.119 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-11T20:15:32.195 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-11T20:15:32.195 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-11T20:15:32.195 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-11T20:15:32.195 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-11T20:15:32.195 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-11T20:15:32.195 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2023-12-11T20:15:32.195 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-11T20:15:32.195 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-11T20:15:32.196 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-11T20:15:32.196 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-11T20:15:32.687 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2023-12-11T20:15:32.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2023-12-11T20:15:32.689 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-11T20:15:32.808 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-11T20:15:32.844 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:32.946 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:32.947+0000 7f6e7456e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-11T20:15:32.946 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:32.947+0000 7f6e7456e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-11T20:15:32.946 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:32.947+0000 7f6e7456e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-11T20:15:32.946 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:32.947+0000 7f6e7456e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-11T20:15:32.947 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:32.947+0000 7f6e7456e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-11T20:15:32.947 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:32.948+0000 7f6e7456e500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-11T20:15:35.788 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-11T20:15:35.893 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:35.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:35.989+0000 7f428e838500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-11T20:15:35.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:35.990+0000 7f428e838500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-11T20:15:35.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:35.990+0000 7f428e838500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-11T20:15:35.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:35.990+0000 7f428e838500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-11T20:15:35.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:35.990+0000 7f428e838500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-11T20:15:35.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:35.990+0000 7f428e838500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-11T20:15:38.830 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-11T20:15:38.861 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:38.956 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:38.957+0000 7f86f5016500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-11T20:15:38.956 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:38.957+0000 7f86f5016500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-11T20:15:38.956 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:38.957+0000 7f86f5016500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-11T20:15:38.956 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:38.957+0000 7f86f5016500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-11T20:15:38.956 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:38.957+0000 7f86f5016500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-11T20:15:38.957 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:38.958+0000 7f86f5016500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-11T20:15:41.806 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-11T20:15:41.837 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:41.928 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:41.929+0000 7f1d3a02a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-11T20:15:41.928 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:41.929+0000 7f1d3a02a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-11T20:15:41.928 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:41.929+0000 7f1d3a02a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-11T20:15:41.928 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:41.929+0000 7f1d3a02a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-11T20:15:41.928 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:41.929+0000 7f1d3a02a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-11T20:15:41.929 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:15:41.930+0000 7f1d3a02a500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-11T20:15:44.774 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-11T20:15:44.806 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-11T20:15:44.806 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:44.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-11T20:15:44.839 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:44.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-11T20:15:44.908 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:44.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-11T20:15:44.978 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:44.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-11T20:15:45.047 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:45.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-11T20:15:45.114 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:45.115 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-11T20:15:45.146 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:45.146 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-11T20:15:45.217 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:45.217 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-11T20:15:45.286 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:45.287 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-11T20:15:45.356 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:45.356 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-11T20:15:45.424 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:15:45.424 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-11T20:15:45.441 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:15:45.441 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-11T20:15:45.479 INFO:tasks.ceph:Adding keys to all mons... 2023-12-11T20:15:45.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-11T20:15:45.483 DEBUG:teuthology.orchestra.run.smithi133:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.y] 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDSbXdlvS9eHBAAiCTZq0wmMAx2SPER71gBFA== 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDVbXdl3TxcORAA96IqVl5zSfMXiPjMz0P7tQ== 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDYbXdlQlhSOBAAi+Uc2tvMvZ8dOu4DSMt+bg== 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDbbXdl69BaNhAA7QtwsbBMSgiL8U+JvROrSg== 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDebXdlaXBCNBAAJ7O4aa6PoHKpBmwtPuemdg== 2023-12-11T20:15:45.524 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDSbXdlQXmjIBAAH01NHRqcfyupp0xECRBmSg== 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDkbXdl/C+JOBAAhtrMGJ8gB+lLOw8MLKm+vQ== 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDnbXdl6FQLOxAAthrQKH4CYJQK3doj/obbRg== 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.6] 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDqbXdl0gAgORAA1fFbu+g9uc16euUMC/XjbQ== 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.7] 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDtbXdluO51NxAAVqdQ64TpTAhZHI9la/ajMg== 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDSbXdlRQdEIxAAHn6VwIo4M9CVssmjOLDBnQ== 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2023-12-11T20:15:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDSbXdlDU+tJhAA26NaS0qgTfU7xO0QDFdAQw== 2023-12-11T20:15:45.546 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.y] 2023-12-11T20:15:45.546 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDSbXdlvS9eHBAAiCTZq0wmMAx2SPER71gBFA== 2023-12-11T20:15:45.546 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDVbXdl3TxcORAA96IqVl5zSfMXiPjMz0P7tQ== 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDYbXdlQlhSOBAAi+Uc2tvMvZ8dOu4DSMt+bg== 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDbbXdl69BaNhAA7QtwsbBMSgiL8U+JvROrSg== 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.3] 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDebXdlaXBCNBAAJ7O4aa6PoHKpBmwtPuemdg== 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.x] 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDSbXdlQXmjIBAAH01NHRqcfyupp0xECRBmSg== 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.4] 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDkbXdl/C+JOBAAhtrMGJ8gB+lLOw8MLKm+vQ== 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.5] 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDnbXdl6FQLOxAAthrQKH4CYJQK3doj/obbRg== 2023-12-11T20:15:45.547 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.6] 2023-12-11T20:15:45.548 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDqbXdl0gAgORAA1fFbu+g9uc16euUMC/XjbQ== 2023-12-11T20:15:45.548 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.7] 2023-12-11T20:15:45.548 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDtbXdluO51NxAAVqdQ64TpTAhZHI9la/ajMg== 2023-12-11T20:15:45.548 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2023-12-11T20:15:45.548 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDSbXdlRQdEIxAAHn6VwIo4M9CVssmjOLDBnQ== 2023-12-11T20:15:45.548 INFO:teuthology.orchestra.run.smithi133.stdout:[client.1] 2023-12-11T20:15:45.548 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQDSbXdlDU+tJhAA26NaS0qgTfU7xO0QDFdAQw== 2023-12-11T20:15:45.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-11T20:15:45.569 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-11T20:15:45.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.644 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.698 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.751 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.805 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-11T20:15:45.858 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-11T20:15:45.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.911 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:45.965 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:46.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:46.018 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:46.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:46.072 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-11T20:15:46.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-11T20:15:46.126 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-11T20:15:46.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-11T20:15:46.183 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-11T20:15:46.238 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-11T20:15:46.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-11T20:15:46.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-11T20:15:46.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-11T20:15:46.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-11T20:15:46.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-11T20:15:46.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-11T20:15:46.577 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-11T20:15:46.604 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-11T20:15:46.702 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-11T20:15:46.728 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:46.731 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-11T20:15:46.783 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-11T20:15:46.783 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-11T20:15:46.783 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-11T20:15:46.784 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-11T20:15:46.786 INFO:tasks.ceph.mon.a:Started 2023-12-11T20:15:46.786 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-11T20:15:46.786 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-11T20:15:46.786 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-11T20:15:46.788 INFO:tasks.ceph.mon.c:Started 2023-12-11T20:15:46.788 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-11T20:15:46.788 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-11T20:15:46.789 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-11T20:15:46.826 INFO:tasks.ceph.mon.b:Started 2023-12-11T20:15:46.826 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-11T20:15:46.827 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-11T20:15:46.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-12-11T20:15:46.829 INFO:tasks.ceph.mgr.y:Started 2023-12-11T20:15:46.829 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-11T20:15:46.829 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-12-11T20:15:46.833 INFO:tasks.ceph.mgr.x:Started 2023-12-11T20:15:46.833 INFO:tasks.ceph:Setting crush tunables to default 2023-12-11T20:15:46.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd crush tunables default 2023-12-11T20:15:46.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.935+0000 7fabd5fc0640 10 monclient: get_monmap_and_config 2023-12-11T20:15:46.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.935+0000 7fabd5fc0640 10 monclient: build_initial_monmap 2023-12-11T20:15:46.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.936+0000 7fabd5fc0640 10 monclient: monmap: 2023-12-11T20:15:46.935 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:15:46.935 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:46.935 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:46.937506+0000 2023-12-11T20:15:46.935 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:46.937506+0000 2023-12-11T20:15:46.935 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:15:46.936 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:15:46.936 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:15:46.936 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:15:46.936 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:15:46.936 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:15:46.936 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:15:46.936 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:15:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.936+0000 7fabd5fc0640 1 Processor -- start 2023-12-11T20:15:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.936+0000 7fabd5fc0640 1 -- start start 2023-12-11T20:15:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.936+0000 7fabd5fc0640 10 monclient: init 2023-12-11T20:15:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 10 monclient: _reopen_session rank -1 2023-12-11T20:15:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-12-11T20:15:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabd00a8990 addr v1:172.21.15.86:6789/0 2023-12-11T20:15:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 0x7fabd01427b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 10 monclient(hunting): picked mon.noname-b con 0x7fabd00a8d70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabd013ef10 addr v1:172.21.15.133:6789/0 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd0142cf0 con 0x7fabd00a8990 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd00aabe0 con 0x7fabd013ef10 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00aaec0 con 0x7fabd00a8d70 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 10 monclient(hunting): _renew_subs 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabd5fc0640 10 monclient(hunting): authenticate will time out at 1433.267546s 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabcf7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabd00a8990 legacy=0x7fabd00b21a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabceffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 msgr2=0x7fabd01427b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2023-12-11T20:15:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabceffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 0x7fabd01427b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-11T20:15:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:46.937+0000 7fabcffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabd013ef10 legacy=0x7fabd013f380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2023-12-11T20:15:47.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:47.138+0000 7fabcf7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabd00a8990 legacy=0x7fabd00b21a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2023-12-11T20:15:47.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:47.138+0000 7fabceffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 msgr2=0x7fabd01427b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2023-12-11T20:15:47.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:47.138+0000 7fabceffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 0x7fabd01427b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-11T20:15:47.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:47.138+0000 7fabcffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabd013ef10 legacy=0x7fabd013f380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2023-12-11T20:15:47.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:47.539+0000 7fabcf7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabd00a8990 legacy=0x7fabd00b21a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2023-12-11T20:15:47.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:47.539+0000 7fabceffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 msgr2=0x7fabd01427b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2023-12-11T20:15:47.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:47.539+0000 7fabceffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 0x7fabd01427b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-11T20:15:47.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:47.539+0000 7fabcffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabd013ef10 legacy=0x7fabd013f380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2023-12-11T20:15:48.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:48.340+0000 7fabcf7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabd00a8990 legacy=0x7fabd00b21a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2023-12-11T20:15:48.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:48.340+0000 7fabceffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 msgr2=0x7fabd01427b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2023-12-11T20:15:48.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:48.340+0000 7fabceffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 0x7fabd01427b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-11T20:15:48.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:48.340+0000 7fabcffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabd013ef10 legacy=0x7fabd013f380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient(hunting): tick 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient(hunting): _check_auth_tickets 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 monclient(hunting): continuing hunt 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 msgr2=0x7fabd01427b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00a8d70 0x7fabd01427b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabd013ef10 legacy=0x7fabd013f380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabd00a8990 legacy=0x7fabd00b21a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient: _add_conns ranks=[4,3] 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 0x7fabbc005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fabbc0036c0 addr v2:172.21.15.133:3300/0 2023-12-11T20:15:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 0x7fabbc008650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fabbc006190 addr v2:172.21.15.86:3300/0 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd00aaec0 con 0x7fabbc006190 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabbc008d00 con 0x7fabbc0036c0 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcd7fa640 10 monclient(hunting): _renew_subs 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabceffd640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 msgr2=0x7fabbc008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabceffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 0x7fabbc008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcf7fe640 1 -- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 msgr2=0x7fabbc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.133:3300/0 2023-12-11T20:15:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:49.937+0000 7fabcf7fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 0x7fabbc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-11T20:15:50.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:50.138+0000 7fabceffd640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 msgr2=0x7fabbc008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2023-12-11T20:15:50.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:50.138+0000 7fabceffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 0x7fabbc008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-11T20:15:50.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:50.138+0000 7fabcf7fe640 1 -- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 msgr2=0x7fabbc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.133:3300/0 2023-12-11T20:15:50.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:50.138+0000 7fabcf7fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 0x7fabbc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-11T20:15:50.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:50.539+0000 7fabceffd640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 msgr2=0x7fabbc008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2023-12-11T20:15:50.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:50.539+0000 7fabceffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 0x7fabbc008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-11T20:15:50.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:50.539+0000 7fabcf7fe640 1 -- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 msgr2=0x7fabbc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.133:3300/0 2023-12-11T20:15:50.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:50.539+0000 7fabcf7fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 0x7fabbc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-11T20:15:51.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:51.340+0000 7fabceffd640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 msgr2=0x7fabbc008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2023-12-11T20:15:51.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:51.340+0000 7fabceffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 0x7fabbc008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-11T20:15:51.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:51.340+0000 7fabcf7fe640 1 -- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 msgr2=0x7fabbc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.133:3300/0 2023-12-11T20:15:51.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:51.340+0000 7fabcf7fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 0x7fabbc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-11T20:15:52.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient(hunting): tick 2023-12-11T20:15:52.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient(hunting): _check_auth_tickets 2023-12-11T20:15:52.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 monclient(hunting): continuing hunt 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 -- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 msgr2=0x7fabbc005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc0036c0 0x7fabbc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 msgr2=0x7fabbc008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc006190 0x7fabbc008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcffff640 1 -- reap_dead start 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabbc003b30 addr v1:172.21.15.86:6789/0 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabbc007d60 addr v1:172.21.15.133:6789/0 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 0x7fabbc00e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:52.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fabbc00bfe0 addr v2:172.21.15.86:3300/0 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd0142260 con 0x7fabbc003b30 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd0142cf0 con 0x7fabbc007d60 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd00aaec0 con 0x7fabbc00bfe0 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcd7fa640 10 monclient(hunting): _renew_subs 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcf7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabbc003b30 legacy=0x7fabbc006640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcffff640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 msgr2=0x7fabbc00e3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabcffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 0x7fabbc00e3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-11T20:15:52.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:52.937+0000 7fabceffd640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabbc007d60 legacy=0x7fabbc008170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2023-12-11T20:15:53.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:53.138+0000 7fabcffff640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 msgr2=0x7fabbc00e3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2023-12-11T20:15:53.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:53.138+0000 7fabcf7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabbc003b30 legacy=0x7fabbc006640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2023-12-11T20:15:53.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:53.138+0000 7fabcffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 0x7fabbc00e3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-11T20:15:53.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:53.138+0000 7fabceffd640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabbc007d60 legacy=0x7fabbc008170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2023-12-11T20:15:53.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:53.539+0000 7fabcffff640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 msgr2=0x7fabbc00e3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2023-12-11T20:15:53.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:53.539+0000 7fabcf7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabbc003b30 legacy=0x7fabbc006640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2023-12-11T20:15:53.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:53.539+0000 7fabcffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 0x7fabbc00e3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-11T20:15:53.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:53.539+0000 7fabceffd640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabbc007d60 legacy=0x7fabbc008170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2023-12-11T20:15:54.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:54.340+0000 7fabcffff640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 msgr2=0x7fabbc00e3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2023-12-11T20:15:54.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:54.340+0000 7fabcf7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabbc003b30 legacy=0x7fabbc006640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2023-12-11T20:15:54.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:54.340+0000 7fabcffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 0x7fabbc00e3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-11T20:15:54.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:54.340+0000 7fabceffd640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabbc007d60 legacy=0x7fabbc008170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2023-12-11T20:15:54.375 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:00:07.444 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:15:54.637 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:00:07.745 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:15:54.652 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:00:07.761 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:15:54.688 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:00:07.756 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:15:54.964 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:00:08.073 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:15:54.982 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:00:08.090 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:15:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient(hunting): tick 2023-12-11T20:15:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient(hunting): _check_auth_tickets 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 monclient(hunting): continuing hunt 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 msgr2=0x7fabbc00e3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabbc00bfe0 0x7fabbc00e3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7fabbc007d60 legacy=0x7fabbc008170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fabbc003b30 legacy=0x7fabbc006640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient: _add_conns ranks=[2,4] 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fabbc00e940 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fabbc00f3b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd00aaec0 con 0x7fabbc00f3b0 2023-12-11T20:15:55.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:55.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc008d00 con 0x7fabbc00e940 2023-12-11T20:15:55.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcd7fa640 10 monclient(hunting): _renew_subs 2023-12-11T20:15:55.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcf7fe640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 msgr2=0x7fabbc00ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2023-12-11T20:15:55.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:55.937+0000 7fabcf7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-11T20:15:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.102+0000 7fabceffd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:56.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.122+0000 7fabceffd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:36220/0 (socket says 172.21.15.86:36220) 2023-12-11T20:15:56.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.122+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 learned_addr learned my addr 172.21.15.86:0/2831909675 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:15:56.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.122+0000 7fabceffd640 10 monclient(hunting): get_auth_request con 0x7fabbc00f3b0 auth_method 0 2023-12-11T20:15:56.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.122+0000 7fabceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:56.122 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.122+0000 7fabceffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:56.122 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.122+0000 7fabceffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:15:56.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.138+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 msgr2=0x7fabbc00ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2023-12-11T20:15:56.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.138+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-11T20:15:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.157+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-11T20:15:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.157+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:15:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.157+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:15:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.157+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-11T20:15:56.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.358+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:56.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.359+0000 7fabceffd640 10 monclient(hunting): get_auth_request con 0x7fabbc00f3b0 auth_method 0 2023-12-11T20:15:56.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.359+0000 7fabceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:56.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.359+0000 7fabceffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:56.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.359+0000 7fabceffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:15:56.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.359+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-11T20:15:56.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.359+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:15:56.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.359+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:15:56.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.360+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-11T20:15:56.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.690+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:56.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.722+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request con 0x7fabbc00e940 auth_method 0 2023-12-11T20:15:56.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.722+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:56.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.722+0000 7fabcf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:56.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.722+0000 7fabcf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:15:56.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.724+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 msgr2=0x7fabbc00ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-11T20:15:56.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.724+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 msgr2=0x7fabbc00ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:15:56.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.724+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:15:56.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:56.724+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-11T20:15:57.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.039+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:57.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.040+0000 7fabceffd640 10 monclient(hunting): get_auth_request con 0x7fabbc00f3b0 auth_method 0 2023-12-11T20:15:57.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.040+0000 7fabceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:57.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.040+0000 7fabceffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:57.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.040+0000 7fabceffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:15:57.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.041+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-11T20:15:57.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.041+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:15:57.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.041+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:15:57.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.041+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-11T20:15:57.556 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.557+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:57.557 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.558+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request con 0x7fabbc00e940 auth_method 0 2023-12-11T20:15:57.557 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.558+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:57.557 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.558+0000 7fabcf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:57.557 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.558+0000 7fabcf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:15:57.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.559+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 msgr2=0x7fabbc00ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-11T20:15:57.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.559+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 msgr2=0x7fabbc00ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:15:57.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.559+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:15:57.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.559+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-11T20:15:57.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.843+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:57.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.843+0000 7fabceffd640 10 monclient(hunting): get_auth_request con 0x7fabbc00f3b0 auth_method 0 2023-12-11T20:15:57.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.843+0000 7fabceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:57.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.843+0000 7fabceffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:57.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.843+0000 7fabceffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:15:57.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.844+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-11T20:15:57.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.844+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:15:57.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.844+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:15:57.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:57.844+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-11T20:15:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.937+0000 7fabcd7fa640 10 monclient(hunting): tick 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.937+0000 7fabcd7fa640 10 monclient(hunting): _check_auth_tickets 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.937+0000 7fabcd7fa640 1 monclient(hunting): continuing hunt 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.937+0000 7fabcd7fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 msgr2=0x7fabbc00ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabbc00e940 0x7fabbc00ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 msgr2=0x7fabbc00f7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.133:3300/0 conn(0x7fabbc00f3b0 0x7fabbc00f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcffff640 1 -- 172.21.15.86:0/2831909675 reap_dead start 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 0x7fabbc00f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fabbc00ed20 addr v2:172.21.15.86:3300/0 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabbc00ff60 addr v1:172.21.15.86:6789/0 2023-12-11T20:15:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabbc005b00 addr v1:172.21.15.133:6789/0 2023-12-11T20:15:58.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 1 -- 172.21.15.86:0/2831909675 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd00aabe0 con 0x7fabbc00ff60 2023-12-11T20:15:58.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 1 -- 172.21.15.86:0/2831909675 --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd0142260 con 0x7fabbc005b00 2023-12-11T20:15:58.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:58.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 1 -- 172.21.15.86:0/2831909675 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd00aaec0 con 0x7fabbc00ed20 2023-12-11T20:15:58.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.938+0000 7fabcd7fa640 10 monclient(hunting): _renew_subs 2023-12-11T20:15:58.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.995+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 0x7fabbc00f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:58.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.996+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request con 0x7fabbc00ed20 auth_method 0 2023-12-11T20:15:58.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.996+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:58.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.996+0000 7fabcf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:58.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.996+0000 7fabcf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:15:58.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.997+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 msgr2=0x7fabbc00f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-11T20:15:58.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.997+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 msgr2=0x7fabbc00f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:15:58.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.997+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 0x7fabbc00f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:15:58.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:58.997+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 0x7fabbc00f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-11T20:15:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.206+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 0x7fabbc00f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.207+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request con 0x7fabbc00ed20 auth_method 0 2023-12-11T20:15:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.207+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.207+0000 7fabcf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.207+0000 7fabcf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:15:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.380+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 msgr2=0x7fabbc00f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-11T20:15:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.380+0000 7fabcf7fe640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 msgr2=0x7fabbc00f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:15:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.380+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 0x7fabbc00f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:15:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.380+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 0x7fabbc00f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-11T20:15:59.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.646+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749238923 0 0) 0x7fabd00aabe0 con 0x7fabbc00ff60 2023-12-11T20:15:59.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.646+0000 7fabce7fc640 10 monclient(hunting): _negotiate 2023-12-11T20:15:59.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.646+0000 7fabce7fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.646+0000 7fabce7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:15:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.646+0000 7fabce7fc640 10 monclient(hunting): my global_id is 4098 2023-12-11T20:15:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.646+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb8003520 con 0x7fabbc00ff60 2023-12-11T20:15:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.675+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871056280 0 0) 0x7fabd0142260 con 0x7fabbc005b00 2023-12-11T20:15:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.675+0000 7fabce7fc640 10 monclient(hunting): _negotiate 2023-12-11T20:15:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.675+0000 7fabce7fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.675+0000 7fabce7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:15:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.675+0000 7fabce7fc640 10 monclient(hunting): my global_id is 4102 2023-12-11T20:15:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.675+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd00aabe0 con 0x7fabbc005b00 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 939148964 0 0) 0x7fabb8003520 con 0x7fabbc00ff60 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 msgr2=0x7fabbc00f790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 --2- 172.21.15.86:0/2831909675 >> v2:172.21.15.86:3300/0 conn(0x7fabbc00ed20 0x7fabbc00f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 >> v1:172.21.15.133:6789/0 conn(0x7fabbc005b00 legacy=0x7fabbc00c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: _finish_hunting 0 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 monclient: found mon.noname-a-legacy 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2023-12-11T20:15:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd0142cf0 con 0x7fabbc00ff60 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: _finish_auth 0 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1416536749 0 0) 0x7fabc4003cf0 con 0x7fabbc00ff60 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: dump: 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:15:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: _reopen_session rank -1 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 >> v1:172.21.15.86:6789/0 conn(0x7fabbc00ff60 legacy=0x7fabbc008f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabb8003b00 0x7fabb8003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient(hunting): picked mon.a con 0x7fabb8003b00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabb80047c0 0x7fabb8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient(hunting): picked mon.c con 0x7fabb80047c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:15:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80051d0 0x7fabb800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient(hunting): picked mon.b con 0x7fabb80051d0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00aaec0 con 0x7fabb8003b00 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb800c100 con 0x7fabb80051d0 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb800c2c0 con 0x7fabb80047c0 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient(hunting): _renew_subs 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.695+0000 7fabce7fc640 10 monclient(hunting): _finish_auth 0 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.696+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabb80047c0 0x7fabb8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.696+0000 7fabceffd640 10 monclient(hunting): get_auth_request con 0x7fabb80047c0 auth_method 0 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.696+0000 7fabceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.696+0000 7fabceffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.696+0000 7fabceffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:15:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.702+0000 7fabceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:15:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.702+0000 7fabceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:15:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.702+0000 7fabceffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:15:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.720+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabb8003b00 0x7fabb8003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.720+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request con 0x7fabb8003b00 auth_method 0 2023-12-11T20:15:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.720+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.720+0000 7fabcf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.720+0000 7fabcf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:15:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.741+0000 7fabcffff640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80051d0 0x7fabb800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.742+0000 7fabcffff640 10 monclient(hunting): get_auth_request con 0x7fabb80051d0 auth_method 0 2023-12-11T20:15:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.742+0000 7fabcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.742+0000 7fabcffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.742+0000 7fabcffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:15:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.743+0000 7fabcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:15:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.743+0000 7fabcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:15:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.743+0000 7fabcffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80051d0 msgr2=0x7fabb800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80051d0 0x7fabb800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabb8003b00 msgr2=0x7fabb8003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabb8003b00 0x7fabb8003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 10 monclient: _finish_hunting 0 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 1 monclient: found mon.c 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:15:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 1 -- 172.21.15.86:0/2831909675 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb8003520 con 0x7fabb80047c0 2023-12-11T20:15:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 10 monclient: _finish_auth 0 2023-12-11T20:15:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 10 monclient: _check_auth_tickets 2023-12-11T20:15:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:15:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabcffff640 1 -- 172.21.15.86:0/2831909675 reap_dead start 2023-12-11T20:15:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabd5fc0640 5 monclient: authenticate success, global_id 4098 2023-12-11T20:15:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.751+0000 7fabd5fc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:15:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.755+0000 7fabceffd640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabb80047c0 0x7fabb8004c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fabc4000c00 tx=0x7fabb80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa5b823912198b69 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:15:59.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc402f070 con 0x7fabb80047c0 2023-12-11T20:15:59.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 10 monclient: dump: 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:15:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc4035400 con 0x7fabb80047c0 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabd5fc0640 10 monclient: get_monmap_and_config success 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 1 -- 172.21.15.86:0/2831909675 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc40357f0 con 0x7fabb80047c0 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabd5fc0640 4 set_mon_vals no callback set 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.802+0000 7fabce7fc640 10 monclient: dump: 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:15:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:15:59.804 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:15:59.804 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 10 monclient: shutdown 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 1 -- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabb80047c0 msgr2=0x7fabb8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabb80047c0 0x7fabb8004c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fabc4000c00 tx=0x7fabb80057f0 comp rx=0 tx=0).stop 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 1 -- 172.21.15.86:0/2831909675 shutdown_connections 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 1 --2- 172.21.15.86:0/2831909675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabb80047c0 0x7fabb8004c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 1 -- 172.21.15.86:0/2831909675 >> 172.21.15.86:0/2831909675 conn(0x7fabd00b8310 msgr2=0x7fabd00b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 1 -- 172.21.15.86:0/2831909675 shutdown_connections 2023-12-11T20:15:59.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.803+0000 7fabd5fc0640 1 -- 172.21.15.86:0/2831909675 wait complete. 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 10 monclient: build_initial_monmap 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 10 monclient: monmap: 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 1 librados: starting msgr at 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 1 librados: starting objecter 2023-12-11T20:15:59.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 1 Processor -- start 2023-12-11T20:15:59.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 1 -- start start 2023-12-11T20:15:59.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 1 librados: setting wanted keys 2023-12-11T20:15:59.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 1 librados: calling monclient init 2023-12-11T20:15:59.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.804+0000 7fabd5fc0640 10 monclient: init 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient: _reopen_session rank -1 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80047c0 0x7fabd013a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient(hunting): picked mon.noname-b con 0x7fabb80047c0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd013ae30 0x7fabd01da160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient(hunting): picked mon.noname-a con 0x7fabd013ae30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd01da6a0 0x7fabd0140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient(hunting): picked mon.noname-c con 0x7fabd01da6a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0140ae0 con 0x7fabd013ae30 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00ac000 con 0x7fabb80047c0 2023-12-11T20:15:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:15:59.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00b12a0 con 0x7fabd01da6a0 2023-12-11T20:15:59.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient(hunting): _renew_subs 2023-12-11T20:15:59.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.805+0000 7fabd5fc0640 10 monclient(hunting): authenticate will time out at 1446.135308s 2023-12-11T20:15:59.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.813+0000 7fabcffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd01da6a0 0x7fabd0140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:59.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.822+0000 7fabcf7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80047c0 0x7fabd013a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:59.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.822+0000 7fabcf7fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80047c0 0x7fabd013a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:42832/0 (socket says 172.21.15.86:42832) 2023-12-11T20:15:59.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.822+0000 7fabcf7fe640 1 -- 172.21.15.86:0/3290368945 learned_addr learned my addr 172.21.15.86:0/3290368945 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:15:59.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.822+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request con 0x7fabb80047c0 auth_method 0 2023-12-11T20:15:59.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.822+0000 7fabcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:59.822 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.822+0000 7fabcf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.822 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.822+0000 7fabcf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:15:59.822 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.823+0000 7fabcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:15:59.822 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.823+0000 7fabcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:15:59.822 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.823+0000 7fabcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:15:59.824 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.824+0000 7fabcffff640 10 monclient(hunting): get_auth_request con 0x7fabd01da6a0 auth_method 0 2023-12-11T20:15:59.824 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.824+0000 7fabcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:59.824 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.824+0000 7fabcffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.824 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.824+0000 7fabcffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:15:59.824 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.826+0000 7fabcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:15:59.824 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.826+0000 7fabcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:15:59.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.826+0000 7fabcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:15:59.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.829+0000 7fabceffd640 1 --2- 172.21.15.86:0/3290368945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd013ae30 0x7fabd01da160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:15:59.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.830+0000 7fabceffd640 10 monclient(hunting): get_auth_request con 0x7fabd013ae30 auth_method 0 2023-12-11T20:15:59.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.830+0000 7fabceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:15:59.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.830+0000 7fabceffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:15:59.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.830+0000 7fabceffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:15:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2023-12-11T20:15:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 1 -- 172.21.15.86:0/3290368945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd01da6a0 msgr2=0x7fabd0140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:15:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/3290368945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd01da6a0 0x7fabd0140f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 1 -- 172.21.15.86:0/3290368945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd013ae30 msgr2=0x7fabd01da160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:15:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/3290368945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd013ae30 0x7fabd01da160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:15:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 10 monclient: _finish_hunting 0 2023-12-11T20:15:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 1 monclient: found mon.noname-b 2023-12-11T20:15:59.836 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:15:59.836 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:15:59.836 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.836+0000 7fabcf7fe640 1 -- 172.21.15.86:0/3290368945 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd013b290 con 0x7fabb80047c0 2023-12-11T20:15:59.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.839+0000 7fabcf7fe640 1 --2- 172.21.15.86:0/3290368945 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80047c0 0x7fabd013a8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fabc000cdd0 tx=0x7fabc0005480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:15:59.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.839+0000 7fabb7fff640 10 client.?.objecter ms_handle_connect 0x7fabb80047c0 2023-12-11T20:15:59.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.839+0000 7fabb7fff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:15:59.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.884+0000 7fabb7fff640 1 -- 172.21.15.86:0/3290368945 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc000c2e0 con 0x7fabb80047c0 2023-12-11T20:15:59.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:15:59.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:15:59.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: dump: 2023-12-11T20:15:59.884 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: _finish_auth 0 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: _check_auth_tickets 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 1 -- 172.21.15.86:0/3290368945 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc0011040 con 0x7fabb80047c0 2023-12-11T20:15:59.885 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 5 monclient: authenticate success, global_id 4108 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 10 monclient: _renew_subs 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd01dab00 con 0x7fabb80047c0 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 1 -- 172.21.15.86:0/3290368945 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0005b90 con 0x7fabb80047c0 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabb7fff640 10 monclient: dump: 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:15:59.886 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabcdffb640 4 set_mon_vals no callback set 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 10 monclient: _renew_subs 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd0141620 con 0x7fabb80047c0 2023-12-11T20:15:59.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.885+0000 7fabd5fc0640 1 librados: init done 2023-12-11T20:15:59.888 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.886+0000 7fabd5fc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:15:59.888 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.886+0000 7fabd5fc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:15:59.888 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.886+0000 7fabd5fc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:15:59.888 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:15:59.886+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd004e840 con 0x7fabb80047c0 2023-12-11T20:16:00.064 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.065+0000 7fabb7fff640 1 -- 172.21.15.86:0/3290368945 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fabc0021460 con 0x7fabb80047c0 2023-12-11T20:16:00.071 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.071+0000 7fabb7fff640 1 -- 172.21.15.86:0/3290368945 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fabc000a470 con 0x7fabb80047c0 2023-12-11T20:16:00.071 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.071+0000 7fabb7fff640 10 client.4108.objecter ms_dispatch 0x7fabd0000b90 osd_map(1..1 src has 1..1) v4 2023-12-11T20:16:00.071 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.071+0000 7fabb7fff640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2023-12-11T20:16:00.071 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.071+0000 7fabb7fff640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2023-12-11T20:16:00.071 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.072+0000 7fabb7fff640 20 client.4108.objecter dump_active .. 0 homeless 2023-12-11T20:16:00.071 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.072+0000 7fabb7fff640 1 -- 172.21.15.86:0/3290368945 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fabc0021850 con 0x7fabb80047c0 2023-12-11T20:16:00.071 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.072+0000 7fabb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:00.071 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.072+0000 7fabb7fff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:00.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.139+0000 7fabd5fc0640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-11T20:16:00.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.139+0000 7fabd5fc0640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-11T20:16:00.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.139+0000 7fabd5fc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:00.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.139+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fabd01dacf0 con 0x7fabb80047c0 2023-12-11T20:16:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.444+0000 7fabb7fff640 1 -- 172.21.15.86:0/3290368945 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fabc00219f0 con 0x7fabb80047c0 2023-12-11T20:16:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.444+0000 7fabb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-11T20:16:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.444+0000 7fabb7fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-12-11T20:16:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2023-12-11T20:16:00.445 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 10 librados: watch_flush enter 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 10 librados: watch_flush exit 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 20 client.4108.objecter shutdown clearing up homeless session... 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 10 client.4108.objecter successfully canceled tick 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 10 monclient: shutdown 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80047c0 msgr2=0x7fabd013a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 1 --2- 172.21.15.86:0/3290368945 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80047c0 0x7fabd013a8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fabc000cdd0 tx=0x7fabc0005480 comp rx=0 tx=0).stop 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 shutdown_connections 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 1 --2- 172.21.15.86:0/3290368945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd01da6a0 0x7fabd0140f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 1 --2- 172.21.15.86:0/3290368945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd013ae30 0x7fabd01da160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 1 --2- 172.21.15.86:0/3290368945 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabb80047c0 0x7fabd013a8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:00.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.445+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 >> 172.21.15.86:0/3290368945 conn(0x7fabd00b8310 msgr2=0x7fabd00b6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:00.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.446+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 shutdown_connections 2023-12-11T20:16:00.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.446+0000 7fabd5fc0640 1 -- 172.21.15.86:0/3290368945 wait complete. 2023-12-11T20:16:00.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:00.446+0000 7fabd5fc0640 1 librados: shutdown 2023-12-11T20:16:00.447 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:00.446+0000 7f07fb213040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-11T20:16:00.456 INFO:tasks.ceph:check_enable_crimson: False 2023-12-11T20:16:00.456 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-11T20:16:00.457 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:16:00.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-11T20:16:00.483 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:16:00.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-11T20:16:00.500 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:00.501+0000 7f07fb213040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-11T20:16:00.514 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:00.516+0000 7fe472032040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-11T20:16:00.527 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:16:00.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-11T20:16:00.566 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:00.567+0000 7fe472032040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-11T20:16:00.595 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:16:00.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-11T20:16:00.663 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:16:00.663 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-11T20:16:00.690 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:16:00.690 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-11T20:16:00.757 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:16:00.757 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-11T20:16:00.825 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-12-11T20:16:00.826 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-11T20:16:00.893 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new feaf0ee1-5c9d-4980-bb27-27c376912955 0 2023-12-11T20:16:01.037 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.035+0000 7f7c3004f640 10 monclient: get_monmap_and_config 2023-12-11T20:16:01.038 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.035+0000 7f7c3004f640 10 monclient: build_initial_monmap 2023-12-11T20:16:01.038 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.035+0000 7f7c3004f640 10 monclient: monmap: 2023-12-11T20:16:01.038 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:01.038 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.038 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:16:01.036518+0000 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:16:01.036518+0000 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.036+0000 7f7c3004f640 1 Processor -- start 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.036+0000 7f7c3004f640 1 -- start start 2023-12-11T20:16:01.039 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.036+0000 7f7c3004f640 10 monclient: init 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.036+0000 7f7c3004f640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.036+0000 7f7c3004f640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c280a8930 0x7f7c280a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c280a8930 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c280b21a0 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f7c2813f0b0 0x7f7c2813f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c2813f0b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280aa980 con 0x7f7c280b21a0 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2813fa20 con 0x7f7c280a8930 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c280aabf0 con 0x7f7c2813f0b0 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c3004f640 10 monclient(hunting): authenticate will time out at 1454.385339s 2023-12-11T20:16:01.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c2e5ec640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f7c2813f0b0 0x7f7c2813f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.037+0000 7f7c2ddeb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c280a8930 0x7f7c280a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c280a8930 0x7f7c280a8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.133:35800/0 (socket says 172.21.15.133:35800) 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2733586955 learned_addr learned my addr 172.21.15.133:0/2733586955 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 10 monclient(hunting): get_auth_request con 0x7f7c280a8930 auth_method 0 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2e5ec640 10 monclient(hunting): get_auth_request con 0x7f7c2813f0b0 auth_method 0 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2e5ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2e5ec640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2e5ec640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.038+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.042 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.039+0000 7f7c2e5ec640 1 -- 172.21.15.133:0/2733586955 >> v2:172.21.15.133:3300/0 conn(0x7f7c2813f0b0 msgr2=0x7f7c2813f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-11T20:16:01.042 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.039+0000 7f7c2e5ec640 1 -- 172.21.15.133:0/2733586955 >> v2:172.21.15.133:3300/0 conn(0x7f7c2813f0b0 msgr2=0x7f7c2813f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-11T20:16:01.042 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.039+0000 7f7c2e5ec640 1 --2- 172.21.15.133:0/2733586955 >> v2:172.21.15.133:3300/0 conn(0x7f7c2813f0b0 0x7f7c2813f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:16:01.042 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.039+0000 7f7c2e5ec640 1 --2- 172.21.15.133:0/2733586955 >> v2:172.21.15.133:3300/0 conn(0x7f7c2813f0b0 0x7f7c2813f4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2733586955 >> v2:172.21.15.133:3300/0 conn(0x7f7c2813f0b0 msgr2=0x7f7c2813f4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2733586955 >> v2:172.21.15.133:3300/0 conn(0x7f7c2813f0b0 0x7f7c2813f4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2733586955 >> v1:172.21.15.133:6789/0 conn(0x7f7c280b21a0 legacy=0x7f7c281424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 10 monclient: _finish_hunting 0 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 1 monclient: found mon.noname-a 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.040+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2733586955 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c280aaed0 con 0x7f7c280a8930 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.042+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c280a8930 0x7f7c280a8d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1e40e971f8fcb6 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c280a8930 msgr2=0x7f7c280a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c280a8930 msgr2=0x7f7c280a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-11T20:16:01.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c280a8930 0x7f7c280a8d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c1802b910 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c280a8930 0x7f7c280a8d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c1802b910 comp rx=0 tx=0).stop 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 1 -- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c280a8930 msgr2=0x7f7c280a8d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c280a8930 0x7f7c280a8d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 10 monclient: _add_conns ranks=[2,0] 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c14003500 0x7f7c14005950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 10 monclient(hunting): picked mon.noname-b con 0x7f7c14003500 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c14006220 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 1 -- 172.21.15.133:0/2733586955 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1400a190 con 0x7f7c14006220 2023-12-11T20:16:01.045 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 1 -- 172.21.15.133:0/2733586955 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1400b350 con 0x7f7c14003500 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.043+0000 7f7c2cde9640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.044+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c14003500 0x7f7c14005950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.044+0000 7f7c2ddeb640 10 monclient(hunting): get_auth_request con 0x7f7c14003500 auth_method 0 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.044+0000 7f7c2ddeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.044+0000 7f7c2ddeb640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.044+0000 7f7c2ddeb640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.045+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.045+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.046 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.045+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:16:01.047 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.047+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_done global_id 4113 payload 293 2023-12-11T20:16:01.047 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.047+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2733586955 >> v1:172.21.15.86:6789/0 conn(0x7f7c14006220 legacy=0x7f7c140086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:01.047 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.047+0000 7f7c2ddeb640 10 monclient: _finish_hunting 0 2023-12-11T20:16:01.047 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.047+0000 7f7c2ddeb640 1 monclient: found mon.noname-b 2023-12-11T20:16:01.047 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.047+0000 7f7c2ddeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:01.047 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.047+0000 7f7c2ddeb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:01.047 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.047+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2733586955 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1400b5d0 con 0x7f7c14003500 2023-12-11T20:16:01.047 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.049+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c14003500 0x7f7c14005950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7c280b4150 tx=0x7f7c18009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97f5c0a12e9eba9a server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:01.052 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 1 -- 172.21.15.133:0/2733586955 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18031870 con 0x7f7c14003500 2023-12-11T20:16:01.052 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: dump: 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: _finish_auth 0 2023-12-11T20:16:01.053 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: _check_auth_tickets 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 1 -- 172.21.15.133:0/2733586955 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c180319d0 con 0x7f7c14003500 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c3004f640 5 monclient: authenticate success, global_id 4113 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c3004f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 1 -- 172.21.15.133:0/2733586955 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18031d40 con 0x7f7c14003500 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c2cde9640 10 monclient: dump: 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.054 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.055 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.055 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.055 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.055 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.055 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.055 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c3004f640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:01.055 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.054+0000 7f7c3004f640 4 set_mon_vals no callback set 2023-12-11T20:16:01.059 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 10 monclient: shutdown 2023-12-11T20:16:01.059 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:01.059 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 1 -- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c14003500 msgr2=0x7f7c14005950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.059 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c14003500 0x7f7c14005950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7c280b4150 tx=0x7f7c18009240 comp rx=0 tx=0).stop 2023-12-11T20:16:01.059 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c2e5ec640 1 -- 172.21.15.133:0/2733586955 reap_dead start 2023-12-11T20:16:01.059 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 1 -- 172.21.15.133:0/2733586955 shutdown_connections 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 1 --2- 172.21.15.133:0/2733586955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c280a8930 0x7f7c280a8d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 1 -- 172.21.15.133:0/2733586955 >> 172.21.15.133:0/2733586955 conn(0x7f7c280b82f0 msgr2=0x7f7c280b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 1 -- 172.21.15.133:0/2733586955 shutdown_connections 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 1 -- 172.21.15.133:0/2733586955 wait complete. 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 10 monclient: build_initial_monmap 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.055+0000 7f7c3004f640 10 monclient: monmap: 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.060 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 librados: starting msgr at 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 librados: starting objecter 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 Processor -- start 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 -- start start 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 librados: setting wanted keys 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 librados: calling monclient init 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient: init 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:01.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c280a8930 0x7f7c28137ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c280a8930 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c14003500 0x7f7c28145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c14003500 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c28143190 0x7f7c281d7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c28143190 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c28139480 con 0x7f7c28143190 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281d6e70 con 0x7f7c14003500 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c28145530 con 0x7f7c280a8930 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:01.062 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.056+0000 7f7c3004f640 10 monclient(hunting): authenticate will time out at 1454.405094s 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2ddeb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c280a8930 0x7f7c28137ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2d5ea640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c14003500 0x7f7c28145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2e5ec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c28143190 0x7f7c281d7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2ddeb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c280a8930 0x7f7c28137ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.133:36866/0 (socket says 172.21.15.133:36866) 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2941417281 learned_addr learned my addr 172.21.15.133:0/2941417281 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2ddeb640 10 monclient(hunting): get_auth_request con 0x7f7c280a8930 auth_method 0 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2ddeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2ddeb640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2ddeb640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2d5ea640 10 monclient(hunting): get_auth_request con 0x7f7c14003500 auth_method 0 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2d5ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2d5ea640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2d5ea640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2e5ec640 10 monclient(hunting): get_auth_request con 0x7f7c28143190 auth_method 0 2023-12-11T20:16:01.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2e5ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2e5ec640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.057+0000 7f7c2e5ec640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2d5ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2d5ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2d5ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2e5ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2e5ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.058+0000 7f7c2e5ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2941417281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c14003500 msgr2=0x7f7c28145d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2941417281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c14003500 0x7f7c28145d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2941417281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c28143190 msgr2=0x7f7c281d7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2941417281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c28143190 0x7f7c281d7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 10 monclient: _finish_hunting 0 2023-12-11T20:16:01.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 1 monclient: found mon.noname-c 2023-12-11T20:16:01.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:01.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:01.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.060+0000 7f7c2ddeb640 1 -- 172.21.15.133:0/2941417281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c281d78d0 con 0x7f7c280a8930 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.063+0000 7f7c2ddeb640 1 --2- 172.21.15.133:0/2941417281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c280a8930 0x7f7c28137ed0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.063+0000 7f7c1e7fc640 10 client.?.objecter ms_handle_connect 0x7f7c280a8930 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.063+0000 7f7c1e7fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 1 -- 172.21.15.133:0/2941417281 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1804a040 con 0x7f7c280a8930 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: dump: 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.067 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 1 -- 172.21.15.133:0/2941417281 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c18003a20 con 0x7f7c280a8930 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 1 -- 172.21.15.133:0/2941417281 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18003db0 con 0x7f7c280a8930 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1e7fc640 10 monclient: dump: 2023-12-11T20:16:01.068 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 5 monclient: authenticate success, global_id 4103 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c1ffff640 4 set_mon_vals no callback set 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 10 monclient: _renew_subs 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c281d7a60 con 0x7f7c280a8930 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 10 client.4103.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 10 monclient: _renew_subs 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c281339f0 con 0x7f7c280a8930 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.067+0000 7f7c3004f640 1 librados: init done 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.068+0000 7f7c3004f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.068+0000 7f7c3004f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.068+0000 7f7c3004f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:01.070 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.068+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c281d6cd0 con 0x7f7c280a8930 2023-12-11T20:16:01.072 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.073+0000 7f07fb213040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-11T20:16:01.086 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.088+0000 7f7c1e7fc640 1 -- 172.21.15.133:0/2941417281 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7c18046780 con 0x7f7c280a8930 2023-12-11T20:16:01.086 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.088+0000 7f7c1e7fc640 1 -- 172.21.15.133:0/2941417281 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7c1802d040 con 0x7f7c280a8930 2023-12-11T20:16:01.086 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.088+0000 7f7c1e7fc640 10 client.4103.objecter ms_dispatch 0x7f7c28000b90 osd_map(2..2 src has 1..2) v4 2023-12-11T20:16:01.086 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.088+0000 7f7c1e7fc640 3 client.4103.objecter handle_osd_map got epochs [2,2] > 0 2023-12-11T20:16:01.086 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.088+0000 7f7c1e7fc640 3 client.4103.objecter handle_osd_map decoding full epoch 2 2023-12-11T20:16:01.086 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.088+0000 7f7c1e7fc640 20 client.4103.objecter dump_active .. 0 homeless 2023-12-11T20:16:01.113 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.115+0000 7fe472032040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-11T20:16:01.126 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.127+0000 7f07fb213040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-11T20:16:01.164 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.166+0000 7fe472032040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-11T20:16:01.172 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.173+0000 7f07fb213040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-11T20:16:01.208 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.210+0000 7fe472032040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-11T20:16:01.227 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.228+0000 7f07fb213040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-11T20:16:01.261 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.263+0000 7fe472032040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-11T20:16:01.270 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.271+0000 7f07fb213040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-11T20:16:01.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.277+0000 7f7c1e7fc640 1 -- 172.21.15.133:0/2941417281 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7c281d6cd0 con 0x7f7c280a8930 2023-12-11T20:16:01.276 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.277+0000 7f7c1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:01.276 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.277+0000 7f7c1e7fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:01.304 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.306+0000 7fe472032040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-11T20:16:01.347 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.349+0000 7f7c3004f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "feaf0ee1-5c9d-4980-bb27-27c376912955", "id": 0}] 2023-12-11T20:16:01.348 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.349+0000 7f7c3004f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "feaf0ee1-5c9d-4980-bb27-27c376912955", "id": 0}] 2023-12-11T20:16:01.348 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.349+0000 7f7c3004f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:01.348 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.349+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "feaf0ee1-5c9d-4980-bb27-27c376912955", "id": 0} v 0) v1 -- 0x7f7c28145670 con 0x7f7c280a8930 2023-12-11T20:16:01.378 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.379+0000 7f07fb213040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-11T20:16:01.410 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.411+0000 7fe472032040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-11T20:16:01.452 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.452+0000 7f07fb213040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-11T20:16:01.479 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.481+0000 7fe472032040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-11T20:16:01.501 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.502+0000 7f07fb213040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-11T20:16:01.527 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.528+0000 7fe472032040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-11T20:16:01.557 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.558+0000 7f07fb213040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-11T20:16:01.565 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.566+0000 7f7c1e7fc640 1 -- 172.21.15.133:0/2941417281 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "feaf0ee1-5c9d-4980-bb27-27c376912955", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7c18055070 con 0x7f7c280a8930 2023-12-11T20:16:01.565 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.566+0000 7f7c1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "feaf0ee1-5c9d-4980-bb27-27c376912955", "id": 0}] 2023-12-11T20:16:01.565 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.566+0000 7f7c1e7fc640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:01.565 INFO:teuthology.orchestra.run.smithi133.stdout:0 2023-12-11T20:16:01.566 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 10 librados: watch_flush enter 2023-12-11T20:16:01.566 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 10 librados: watch_flush exit 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 20 client.4103.objecter shutdown clearing up homeless session... 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 10 client.4103.objecter successfully canceled tick 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 10 monclient: shutdown 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c280a8930 msgr2=0x7f7c28137ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 --2- 172.21.15.133:0/2941417281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c280a8930 0x7f7c28137ed0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c180046c0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 shutdown_connections 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 --2- 172.21.15.133:0/2941417281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c28143190 0x7f7c281d7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 --2- 172.21.15.133:0/2941417281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c14003500 0x7f7c28145d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 --2- 172.21.15.133:0/2941417281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c280a8930 0x7f7c28137ed0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 >> 172.21.15.133:0/2941417281 conn(0x7f7c280b82f0 msgr2=0x7f7c280b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 shutdown_connections 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 -- 172.21.15.133:0/2941417281 wait complete. 2023-12-11T20:16:01.567 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.567+0000 7f7c3004f640 1 librados: shutdown 2023-12-11T20:16:01.576 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new f605c3b7-5dc2-4b26-a95e-eff31a2ab106 1 2023-12-11T20:16:01.581 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.583+0000 7fe472032040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-11T20:16:01.605 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.606+0000 7f07fb213040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-11T20:16:01.631 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:01.633+0000 7fe472032040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-11T20:16:01.681 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.679+0000 7f552bba4640 10 monclient: get_monmap_and_config 2023-12-11T20:16:01.681 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.679+0000 7f552bba4640 10 monclient: build_initial_monmap 2023-12-11T20:16:01.681 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.679+0000 7f552bba4640 10 monclient: monmap: 2023-12-11T20:16:01.681 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:01.681 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:16:01.680032+0000 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:16:01.680032+0000 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:01.682 INFO:teuthology.orchestra.run.smithi133.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.679+0000 7f552bba4640 1 Processor -- start 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.679+0000 7f552bba4640 1 -- start start 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.679+0000 7f552bba4640 10 monclient: init 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f552413bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient(hunting): picked mon.noname-b con 0x7f552406f550 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f552413c430 0x7f55241421b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient(hunting): picked mon.noname-c con 0x7f552413c430 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:01.683 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5524071f10 0x7f55241426f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.684 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient(hunting): picked mon.noname-a con 0x7f5524071f10 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:01.684 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.684 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f552406f930 con 0x7f5524071f10 2023-12-11T20:16:01.684 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.684 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f552406fbe0 con 0x7f552413c430 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5524071810 con 0x7f552406f550 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.680+0000 7f552bba4640 10 monclient(hunting): authenticate will time out at 1455.028862s 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552913f640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f552413c430 0x7f55241421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f5529940640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f552413bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552a141640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5524071f10 0x7f55241426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f5529940640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f552413bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.133:36872/0 (socket says 172.21.15.133:36872) 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f5529940640 1 -- 172.21.15.133:0/3538216986 learned_addr learned my addr 172.21.15.133:0/3538216986 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f5529940640 10 monclient(hunting): get_auth_request con 0x7f552406f550 auth_method 0 2023-12-11T20:16:01.685 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f5529940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f5529940640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f5529940640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552913f640 10 monclient(hunting): get_auth_request con 0x7f552413c430 auth_method 0 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552913f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552913f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552913f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552a141640 10 monclient(hunting): get_auth_request con 0x7f5524071f10 auth_method 0 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552a141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552a141640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.681+0000 7f552a141640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.686 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.687 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.687 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.687 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f552913f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.687 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f552913f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.687 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f552913f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.687 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.687 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.687 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.682+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.688 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2023-12-11T20:16:01.688 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 1 -- 172.21.15.133:0/3538216986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 msgr2=0x7f552413bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:01.689 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 1 --2- 172.21.15.133:0/3538216986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f552413bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.689 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 1 -- 172.21.15.133:0/3538216986 >> v2:172.21.15.86:3300/0 conn(0x7f5524071f10 msgr2=0x7f55241426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:01.689 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 1 --2- 172.21.15.133:0/3538216986 >> v2:172.21.15.86:3300/0 conn(0x7f5524071f10 0x7f55241426f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.689 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 10 monclient: _finish_hunting 0 2023-12-11T20:16:01.689 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 1 monclient: found mon.noname-c 2023-12-11T20:16:01.689 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:01.689 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:01.689 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.683+0000 7f552913f640 1 -- 172.21.15.133:0/3538216986 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5524071af0 con 0x7f552413c430 2023-12-11T20:16:01.690 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.687+0000 7f552913f640 1 --2- 172.21.15.133:0/3538216986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241421b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55240b21e0 tx=0x7f552002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9b565a50176a2f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:01.693 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 1 -- 172.21.15.133:0/3538216986 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552002e070 con 0x7f552413c430 2023-12-11T20:16:01.693 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.693 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:01.693 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: dump: 2023-12-11T20:16:01.693 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.693 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.693 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.693 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.694 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.694 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.694 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.694 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.694 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.694 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.695 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: _finish_auth 0 2023-12-11T20:16:01.695 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: _check_auth_tickets 2023-12-11T20:16:01.695 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:01.695 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 1 -- 172.21.15.133:0/3538216986 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55200315a0 con 0x7f552413c430 2023-12-11T20:16:01.695 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 5 monclient: authenticate success, global_id 4123 2023-12-11T20:16:01.695 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:01.695 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:01.695 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 1 -- 172.21.15.133:0/3538216986 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520031930 con 0x7f552413c430 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 4 set_mon_vals no callback set 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552893e640 10 monclient: dump: 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.696 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.697 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.697 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.697 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.697 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.702 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 10 monclient: shutdown 2023-12-11T20:16:01.702 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 1 -- 172.21.15.133:0/3538216986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 msgr2=0x7f55241421b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 1 --2- 172.21.15.133:0/3538216986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241421b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55240b21e0 tx=0x7f552002b840 comp rx=0 tx=0).stop 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 1 -- 172.21.15.133:0/3538216986 shutdown_connections 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 1 --2- 172.21.15.133:0/3538216986 >> v2:172.21.15.86:3300/0 conn(0x7f5524071f10 0x7f55241426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 1 --2- 172.21.15.133:0/3538216986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241421b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 1 --2- 172.21.15.133:0/3538216986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f552413bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.694+0000 7f552bba4640 1 -- 172.21.15.133:0/3538216986 >> 172.21.15.133:0/3538216986 conn(0x7f552406c100 msgr2=0x7f55240b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.695+0000 7f552bba4640 1 -- 172.21.15.133:0/3538216986 shutdown_connections 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.695+0000 7f552bba4640 1 -- 172.21.15.133:0/3538216986 wait complete. 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.695+0000 7f552bba4640 10 monclient: build_initial_monmap 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.695+0000 7f552bba4640 10 monclient: monmap: 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:01.703 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.696+0000 7f552bba4640 1 librados: starting msgr at 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.696+0000 7f552bba4640 1 librados: starting objecter 2023-12-11T20:16:01.704 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.696+0000 7f552bba4640 1 Processor -- start 2023-12-11T20:16:01.705 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.696+0000 7f552bba4640 1 -- start start 2023-12-11T20:16:01.705 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.696+0000 7f552bba4640 1 librados: setting wanted keys 2023-12-11T20:16:01.705 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.696+0000 7f552bba4640 1 librados: calling monclient init 2023-12-11T20:16:01.705 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.696+0000 7f552bba4640 10 monclient: init 2023-12-11T20:16:01.705 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:01.705 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:16:01.705 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f552413afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.705 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient(hunting): picked mon.noname-c con 0x7f552406f550 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 0x7f5524141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient(hunting): picked mon.noname-a con 0x7f5524071f10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241d5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient(hunting): picked mon.noname-b con 0x7f552413c430 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55240b2060 con 0x7f5524071f10 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55240b3cb0 con 0x7f552413c430 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55240b22b0 con 0x7f552406f550 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552bba4640 10 monclient(hunting): authenticate will time out at 1455.045350s 2023-12-11T20:16:01.706 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f5529940640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f552413afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552a141640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241d5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552913f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 0x7f5524141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552a141640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241d5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:40278/0 (socket says 172.21.15.133:40278) 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.697+0000 7f552a141640 1 -- 172.21.15.133:0/1460392138 learned_addr learned my addr 172.21.15.133:0/1460392138 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552a141640 10 monclient(hunting): get_auth_request con 0x7f552413c430 auth_method 0 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552a141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552a141640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552a141640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.707 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f5529940640 10 monclient(hunting): get_auth_request con 0x7f552406f550 auth_method 0 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f5529940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f5529940640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f5529940640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552913f640 10 monclient(hunting): get_auth_request con 0x7f5524071f10 auth_method 0 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552913f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552913f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552913f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.708 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552913f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552913f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.698+0000 7f552913f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 1 -- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 msgr2=0x7f552413afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f552413afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.709 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 1 -- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 msgr2=0x7f5524141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 0x7f5524141ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 10 monclient: _finish_hunting 0 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 1 monclient: found mon.noname-b 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.701+0000 7f552a141640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5520009400 con 0x7f552413c430 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.703+0000 7f552a141640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241d5f00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f551c00efc0 tx=0x7f551c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.703+0000 7f551a7fc640 10 client.?.objecter ms_handle_connect 0x7f552413c430 2023-12-11T20:16:01.710 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.703+0000 7f551a7fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:01.711 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c014480 con 0x7f552413c430 2023-12-11T20:16:01.711 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.711 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:01.711 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: dump: 2023-12-11T20:16:01.711 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:01.712 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c00f040 con 0x7f552413c430 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f552bba4640 5 monclient: authenticate success, global_id 4126 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f552bba4640 10 monclient: _renew_subs 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f552bba4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f552bba4640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55240b2b50 con 0x7f552413c430 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c014820 con 0x7f552413c430 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:01.713 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551a7fc640 10 monclient: dump: 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.714 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.715 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f552bba4640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:01.715 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f552bba4640 10 monclient: _renew_subs 2023-12-11T20:16:01.715 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.707+0000 7f552bba4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:01.715 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.706+0000 7f551bfff640 4 set_mon_vals no callback set 2023-12-11T20:16:01.715 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.707+0000 7f552bba4640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55241d6440 con 0x7f552413c430 2023-12-11T20:16:01.715 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.707+0000 7f552bba4640 1 librados: init done 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f552a141640 1 -- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 msgr2=0x7f55241d5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f552a141640 1 -- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 msgr2=0x7f55241d5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f552a141640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241d5f00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f551c00efc0 tx=0x7f551c00c490 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f552a141640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241d5f00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f551c00efc0 tx=0x7f551c00c490 comp rx=0 tx=0).stop 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 msgr2=0x7f55241d5f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f552413c430 0x7f55241d5f00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 0x7f5508007250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.719 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient(hunting): picked mon.a con 0x7f5524071f10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5508007c20 0x7f550800a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f552bba4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient(hunting): picked mon.b con 0x7f5508007c20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f550800cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient(hunting): picked mon.c con 0x7f552406f550 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550800d5e0 con 0x7f5524071f10 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550800d720 con 0x7f5508007c20 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550800d860 con 0x7f552406f550 2023-12-11T20:16:01.720 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f551a7fc640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f552bba4640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f552a141640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f550800cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.708+0000 7f5529940640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 0x7f5508007250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f552a141640 10 monclient(hunting): get_auth_request con 0x7f552406f550 auth_method 0 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f552a141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f552a141640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f552a141640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f5529940640 10 monclient(hunting): get_auth_request con 0x7f5524071f10 auth_method 0 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f5529940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f5529940640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f5529940640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.721 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f5529940640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.709+0000 7f552a141640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 10 monclient(hunting): handle_auth_done global_id 4126 payload 293 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 1 -- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 msgr2=0x7f550800cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f550800cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 1 -- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5508007c20 msgr2=0x7f550800a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5508007c20 0x7f550800a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 10 monclient: _finish_hunting 0 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 1 monclient: found mon.a 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:01.722 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f550800db50 con 0x7f5524071f10 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552404e840 con 0x7f5524071f10 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5514011670 con 0x7f5524071f10 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 10 monclient: _finish_auth 0 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 10 monclient: _check_auth_tickets 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.711+0000 7f5529940640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.713+0000 7f5529940640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 0x7f5508007250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5514011860 tx=0x7f551c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.713+0000 7f551a7fc640 10 client.4126.objecter ms_handle_connect 0x7f5524071f10 2023-12-11T20:16:01.723 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.713+0000 7f551a7fc640 10 client.4126.objecter resend_mon_ops 2023-12-11T20:16:01.919 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.921+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514016070 con 0x7f5524071f10 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.921+0000 7f551a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.921+0000 7f551a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.921+0000 7f551a7fc640 10 monclient: dump: 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:01.920 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:01.921 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.921+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55140040f0 con 0x7f5524071f10 2023-12-11T20:16:01.921 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.921+0000 7f551a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:01.921 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.921+0000 7f551b7fe640 4 set_mon_vals no callback set 2023-12-11T20:16:01.925 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5514004280 con 0x7f5524071f10 2023-12-11T20:16:01.925 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.0 v2:172.21.15.86:3300/0 4 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f5514003c30 con 0x7f5524071f10 2023-12-11T20:16:01.925 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 10 client.4126.objecter ms_dispatch 0x7f5524000b90 osd_map(3..3 src has 1..3) v4 2023-12-11T20:16:01.925 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2023-12-11T20:16:01.925 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2023-12-11T20:16:01.926 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 20 client.4126.objecter dump_active .. 0 homeless 2023-12-11T20:16:01.926 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.0 v2:172.21.15.86:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f551400bbb0 con 0x7f5524071f10 2023-12-11T20:16:01.926 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:01.926 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.927+0000 7f551a7fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:01.930 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.931+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f55140108f0 con 0x7f5524071f10 2023-12-11T20:16:01.930 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.931+0000 7f551a7fc640 10 monclient: handle_mon_command_ack 1 not found 2023-12-11T20:16:01.995 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:01.996+0000 7f07fb213040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-11T20:16:01.998 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.999+0000 7f552bba4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f605c3b7-5dc2-4b26-a95e-eff31a2ab106", "id": 1}] 2023-12-11T20:16:01.998 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.999+0000 7f552bba4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f605c3b7-5dc2-4b26-a95e-eff31a2ab106", "id": 1}] 2023-12-11T20:16:01.998 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.999+0000 7f552bba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:01.998 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:01.999+0000 7f552bba4640 1 -- 172.21.15.133:0/1460392138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f605c3b7-5dc2-4b26-a95e-eff31a2ab106", "id": 1} v 0) v1 -- 0x7f55240ae520 con 0x7f5524071f10 2023-12-11T20:16:02.020 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.022+0000 7fe472032040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-11T20:16:02.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.042+0000 7f551a7fc640 1 -- 172.21.15.133:0/1460392138 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f605c3b7-5dc2-4b26-a95e-eff31a2ab106", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5514010a90 con 0x7f5524071f10 2023-12-11T20:16:02.040 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.042+0000 7f551a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f605c3b7-5dc2-4b26-a95e-eff31a2ab106", "id": 1}] 2023-12-11T20:16:02.041 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.042+0000 7f551a7fc640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:02.041 INFO:teuthology.orchestra.run.smithi133.stdout:1 2023-12-11T20:16:02.042 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 10 librados: watch_flush enter 2023-12-11T20:16:02.042 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 10 librados: watch_flush exit 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 20 client.4126.objecter shutdown clearing up homeless session... 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 10 client.4126.objecter successfully canceled tick 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 10 monclient: shutdown 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 -- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 msgr2=0x7f5508007250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 0x7f5508007250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5514011860 tx=0x7f551c00b040 comp rx=0 tx=0).stop 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 -- 172.21.15.133:0/1460392138 shutdown_connections 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552406f550 0x7f550800cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5508007c20 0x7f550800a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 --2- 172.21.15.133:0/1460392138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5524071f10 0x7f5508007250 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 -- 172.21.15.133:0/1460392138 >> 172.21.15.133:0/1460392138 conn(0x7f552406c100 msgr2=0x7f55240af3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 -- 172.21.15.133:0/1460392138 shutdown_connections 2023-12-11T20:16:02.043 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 -- 172.21.15.133:0/1460392138 wait complete. 2023-12-11T20:16:02.044 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.043+0000 7f552bba4640 1 librados: shutdown 2023-12-11T20:16:02.052 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 94937f44-381b-4c32-b869-184ceeb9ced0 2 2023-12-11T20:16:02.157 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:02.158+0000 7f07fb213040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-11T20:16:02.163 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.164+0000 7fe472032040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-11T20:16:02.191 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.193+0000 7fe8293f5640 10 monclient: get_monmap_and_config 2023-12-11T20:16:02.191 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.193+0000 7fe8293f5640 10 monclient: build_initial_monmap 2023-12-11T20:16:02.191 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.193+0000 7fe8293f5640 10 monclient: monmap: 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:16:02.194106+0000 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:16:02.194106+0000 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:02.192 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.193+0000 7fe8293f5640 1 Processor -- start 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.193+0000 7fe8293f5640 1 -- start start 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.193+0000 7fe8293f5640 10 monclient: init 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8240a8930 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe8240b21a0 0x7fe8240a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient(hunting): picked mon.noname-c con 0x7fe8240b21a0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe82413f0b0 0x7fe82413f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.195 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient(hunting): picked mon.noname-a con 0x7fe82413f0b0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8240aa980 con 0x7fe8240a8930 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82413fa20 con 0x7fe82413f0b0 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8240aabf0 con 0x7fe8240b21a0 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.194+0000 7fe8293f5640 10 monclient(hunting): authenticate will time out at 1455.542700s 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8227fc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fe8240b21a0 0x7fe8240a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe82413f0b0 0x7fe82413f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe82413f0b0 0x7fe82413f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.133:35856/0 (socket says 172.21.15.133:35856) 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 1 -- 172.21.15.133:0/1459245955 learned_addr learned my addr 172.21.15.133:0/1459245955 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:02.196 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 10 monclient(hunting): get_auth_request con 0x7fe82413f0b0 auth_method 0 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8227fc640 10 monclient(hunting): get_auth_request con 0x7fe8240b21a0 auth_method 0 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8227fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8227fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.195+0000 7fe8237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.196+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.196+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.197 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.196+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.198 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2023-12-11T20:16:02.198 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 1 -- 172.21.15.133:0/1459245955 >> v2:172.21.15.133:3300/0 conn(0x7fe8240b21a0 msgr2=0x7fe8240a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:02.198 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 1 --2- 172.21.15.133:0/1459245955 >> v2:172.21.15.133:3300/0 conn(0x7fe8240b21a0 0x7fe8240a8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.199 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 1 -- 172.21.15.133:0/1459245955 >> v1:172.21.15.133:6789/0 conn(0x7fe8240a8930 legacy=0x7fe8241424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.199 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 10 monclient: _finish_hunting 0 2023-12-11T20:16:02.199 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 1 monclient: found mon.noname-a 2023-12-11T20:16:02.199 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:02.199 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:02.199 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.197+0000 7fe8237fe640 1 -- 172.21.15.133:0/1459245955 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8240aaed0 con 0x7fe82413f0b0 2023-12-11T20:16:02.199 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.199+0000 7fe8237fe640 1 --2- 172.21.15.133:0/1459245955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe82413f0b0 0x7fe82413f4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009e50 tx=0x7fe81c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4618c750f153f09 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:02.201 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 1 -- 172.21.15.133:0/1459245955 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c02ca80 con 0x7fe82413f0b0 2023-12-11T20:16:02.201 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.201 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:02.201 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: dump: 2023-12-11T20:16:02.201 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.201 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.201 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: _finish_auth 0 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:02.202 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 1 -- 172.21.15.133:0/1459245955 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c02cbe0 con 0x7fe82413f0b0 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 1 -- 172.21.15.133:0/1459245955 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0343f0 con 0x7fe82413f0b0 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe8293f5640 5 monclient: authenticate success, global_id 4125 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe8293f5640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe8293f5640 4 set_mon_vals no callback set 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.202+0000 7fe821ffb640 10 monclient: dump: 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.203 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.204 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.204 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.204 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.204 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 10 monclient: shutdown 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 1 -- 172.21.15.133:0/1459245955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe82413f0b0 msgr2=0x7fe82413f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 1 --2- 172.21.15.133:0/1459245955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe82413f0b0 0x7fe82413f4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009e50 tx=0x7fe81c02bee0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 1 -- 172.21.15.133:0/1459245955 shutdown_connections 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 1 --2- 172.21.15.133:0/1459245955 >> v2:172.21.15.133:3300/0 conn(0x7fe8240b21a0 0x7fe8240a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 1 --2- 172.21.15.133:0/1459245955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe82413f0b0 0x7fe82413f4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 1 -- 172.21.15.133:0/1459245955 >> 172.21.15.133:0/1459245955 conn(0x7fe8240b82f0 msgr2=0x7fe8240b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 1 -- 172.21.15.133:0/1459245955 shutdown_connections 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.203+0000 7fe8293f5640 1 -- 172.21.15.133:0/1459245955 wait complete. 2023-12-11T20:16:02.209 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.204+0000 7fe8293f5640 10 monclient: build_initial_monmap 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.204+0000 7fe8293f5640 10 monclient: monmap: 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.204+0000 7fe8293f5640 1 librados: starting msgr at 2023-12-11T20:16:02.210 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.204+0000 7fe8293f5640 1 librados: starting objecter 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.204+0000 7fe8293f5640 1 Processor -- start 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 -- start start 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 librados: setting wanted keys 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 librados: calling monclient init 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient: init 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe8240a8930 0x7fe824138010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient(hunting): picked mon.noname-b con 0x7fe8240a8930 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8240b21a0 0x7fe82413a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient(hunting): picked mon.noname-a con 0x7fe8240b21a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe82413f0b0 0x7fe82413add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient(hunting): picked mon.noname-c con 0x7fe82413f0b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:02.211 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82413b310 con 0x7fe8240b21a0 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824138d20 con 0x7fe8240a8930 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82413a470 con 0x7fe82413f0b0 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.205+0000 7fe8293f5640 10 monclient(hunting): authenticate will time out at 1455.553816s 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe8240a8930 0x7fe824138010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8227fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8240b21a0 0x7fe82413a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8237fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe82413f0b0 0x7fe82413add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe8240a8930 0x7fe824138010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:40290/0 (socket says 172.21.15.133:40290) 2023-12-11T20:16:02.212 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 1 -- 172.21.15.133:0/3030312991 learned_addr learned my addr 172.21.15.133:0/3030312991 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 10 monclient(hunting): get_auth_request con 0x7fe8240a8930 auth_method 0 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8227fc640 10 monclient(hunting): get_auth_request con 0x7fe8240b21a0 auth_method 0 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8227fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8227fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8237fe640 10 monclient(hunting): get_auth_request con 0x7fe82413f0b0 auth_method 0 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8237fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8237fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.213 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe822ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.206+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.207+0000 7fe8237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.207+0000 7fe8237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.207+0000 7fe8237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 1 -- 172.21.15.133:0/3030312991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe82413f0b0 msgr2=0x7fe82413add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 1 --2- 172.21.15.133:0/3030312991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe82413f0b0 0x7fe82413add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 1 -- 172.21.15.133:0/3030312991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8240b21a0 msgr2=0x7fe82413a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 1 --2- 172.21.15.133:0/3030312991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8240b21a0 0x7fe82413a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.214 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 10 monclient: _finish_hunting 0 2023-12-11T20:16:02.215 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 1 monclient: found mon.noname-b 2023-12-11T20:16:02.215 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:02.215 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.215 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.209+0000 7fe822ffd640 1 -- 172.21.15.133:0/3030312991 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c009b30 con 0x7fe8240a8930 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.211+0000 7fe822ffd640 1 --2- 172.21.15.133:0/3030312991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe8240a8930 0x7fe824138010 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe81800c430 tx=0x7fe81800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.211+0000 7fe80b7fe640 10 client.?.objecter ms_handle_connect 0x7fe8240a8930 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.211+0000 7fe80b7fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 1 -- 172.21.15.133:0/3030312991 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8180052d0 con 0x7fe8240a8930 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: dump: 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.217 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: _finish_auth 0 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: _check_auth_tickets 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 1 -- 172.21.15.133:0/3030312991 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe818005450 con 0x7fe8240a8930 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:02.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 1 -- 172.21.15.133:0/3030312991 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe818007260 con 0x7fe8240a8930 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe80b7fe640 10 monclient: dump: 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.219 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 5 monclient: authenticate success, global_id 4135 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8217fa640 4 set_mon_vals no callback set 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 10 monclient: _renew_subs 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe824146070 con 0x7fe8240a8930 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 10 monclient: _renew_subs 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe824138550 con 0x7fe8240a8930 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.215+0000 7fe8293f5640 1 librados: init done 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.216+0000 7fe8293f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.216+0000 7fe8293f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:02.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.216+0000 7fe8293f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.221 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.216+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe824139390 con 0x7fe8240a8930 2023-12-11T20:16:02.221 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.218+0000 7fe80b7fe640 1 -- 172.21.15.133:0/3030312991 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe8180067d0 con 0x7fe8240a8930 2023-12-11T20:16:02.221 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.218+0000 7fe80b7fe640 1 -- 172.21.15.133:0/3030312991 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe818003d40 con 0x7fe8240a8930 2023-12-11T20:16:02.221 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.218+0000 7fe80b7fe640 10 client.4135.objecter ms_dispatch 0x7fe824000b90 osd_map(4..4 src has 1..4) v4 2023-12-11T20:16:02.221 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.218+0000 7fe80b7fe640 3 client.4135.objecter handle_osd_map got epochs [4,4] > 0 2023-12-11T20:16:02.221 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.218+0000 7fe80b7fe640 3 client.4135.objecter handle_osd_map decoding full epoch 4 2023-12-11T20:16:02.221 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.219+0000 7fe80b7fe640 20 client.4135.objecter dump_active .. 0 homeless 2023-12-11T20:16:02.283 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.285+0000 7fe472032040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-11T20:16:02.316 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.317+0000 7fe80b7fe640 1 -- 172.21.15.133:0/3030312991 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fe818020060 con 0x7fe8240a8930 2023-12-11T20:16:02.316 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.317+0000 7fe80b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:02.316 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.317+0000 7fe80b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:02.330 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.332+0000 7fe472032040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-11T20:16:02.388 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.389+0000 7fe8293f5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "94937f44-381b-4c32-b869-184ceeb9ced0", "id": 2}] 2023-12-11T20:16:02.388 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.389+0000 7fe8293f5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "94937f44-381b-4c32-b869-184ceeb9ced0", "id": 2}] 2023-12-11T20:16:02.388 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.389+0000 7fe8293f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.388 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.389+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd new", "uuid": "94937f44-381b-4c32-b869-184ceeb9ced0", "id": 2} v 0) v1 -- 0x7fe82404e840 con 0x7fe8240a8930 2023-12-11T20:16:02.433 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.435+0000 7fe472032040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-11T20:16:02.461 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.462+0000 7fe80b7fe640 1 -- 172.21.15.133:0/3030312991 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "94937f44-381b-4c32-b869-184ceeb9ced0", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe8180038c0 con 0x7fe8240a8930 2023-12-11T20:16:02.461 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.462+0000 7fe80b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "94937f44-381b-4c32-b869-184ceeb9ced0", "id": 2}] 2023-12-11T20:16:02.461 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.462+0000 7fe80b7fe640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:02.461 INFO:teuthology.orchestra.run.smithi133.stdout:2 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.463+0000 7fe8293f5640 10 librados: watch_flush enter 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.463+0000 7fe8293f5640 10 librados: watch_flush exit 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.463+0000 7fe8293f5640 20 client.4135.objecter shutdown clearing up homeless session... 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.463+0000 7fe8293f5640 10 client.4135.objecter successfully canceled tick 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.463+0000 7fe8293f5640 10 monclient: shutdown 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.463+0000 7fe8293f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.463+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe8240a8930 msgr2=0x7fe824138010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.463+0000 7fe8293f5640 1 --2- 172.21.15.133:0/3030312991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe8240a8930 0x7fe824138010 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe81800c430 tx=0x7fe81800c8f0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.464+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 shutdown_connections 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.464+0000 7fe8293f5640 1 --2- 172.21.15.133:0/3030312991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe82413f0b0 0x7fe82413add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.464+0000 7fe8293f5640 1 --2- 172.21.15.133:0/3030312991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8240b21a0 0x7fe82413a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.464+0000 7fe8293f5640 1 --2- 172.21.15.133:0/3030312991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe8240a8930 0x7fe824138010 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.463 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.464+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 >> 172.21.15.133:0/3030312991 conn(0x7fe8240b82f0 msgr2=0x7fe8241362e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:02.464 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.464+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 shutdown_connections 2023-12-11T20:16:02.464 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.464+0000 7fe8293f5640 1 -- 172.21.15.133:0/3030312991 wait complete. 2023-12-11T20:16:02.464 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.464+0000 7fe8293f5640 1 librados: shutdown 2023-12-11T20:16:02.473 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 73d8e78f-aa7e-47e7-9ce1-5571f6fc0759 3 2023-12-11T20:16:02.535 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.536+0000 7fe472032040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.573+0000 7f229d8cf640 10 monclient: get_monmap_and_config 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.573+0000 7f229d8cf640 10 monclient: build_initial_monmap 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.573+0000 7f229d8cf640 10 monclient: monmap: 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:16:02.574742+0000 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:16:02.574742+0000 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:02.572 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.573 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:02.573 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:02.573 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:02.573 INFO:teuthology.orchestra.run.smithi133.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:02.573 INFO:teuthology.orchestra.run.smithi133.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:02.573 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.575 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.574+0000 7f229d8cf640 1 Processor -- start 2023-12-11T20:16:02.575 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.574+0000 7f229d8cf640 1 -- start start 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.574+0000 7f229d8cf640 10 monclient: init 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f22980a8930 0x7f2298145920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient(hunting): picked mon.noname-c con 0x7f22980a8930 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22980b2210 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a9b90 0x7f229814c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient(hunting): picked mon.noname-b con 0x7f22980a9b90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22980aac10 con 0x7f22980b2210 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2298145490 con 0x7f22980a8930 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22980aae60 con 0x7f22980a9b90 2023-12-11T20:16:02.576 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f229d8cf640 10 monclient(hunting): authenticate will time out at 1455.923456s 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f22977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a9b90 0x7f229814c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.575+0000 7f2296ffd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f22980a8930 0x7f2298145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a9b90 0x7f229814c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.133:36918/0 (socket says 172.21.15.133:36918) 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 1 -- 172.21.15.133:0/240078718 learned_addr learned my addr 172.21.15.133:0/240078718 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 10 monclient(hunting): get_auth_request con 0x7f22980a9b90 auth_method 0 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f2296ffd640 10 monclient(hunting): get_auth_request con 0x7f22980a8930 auth_method 0 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f2296ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.577 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f2296ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.578 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f2296ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.578 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.578 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.578 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f22977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.578 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f2296ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.578 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f2296ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.578 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.576+0000 7f2296ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.579 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:02.578+0000 7f07fb213040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-11T20:16:02.579 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2023-12-11T20:16:02.579 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 1 -- 172.21.15.133:0/240078718 >> v2:172.21.15.133:3300/0 conn(0x7f22980a8930 msgr2=0x7f2298145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:02.579 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 1 --2- 172.21.15.133:0/240078718 >> v2:172.21.15.133:3300/0 conn(0x7f22980a8930 0x7f2298145920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.579 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 1 -- 172.21.15.133:0/240078718 >> v1:172.21.15.133:6789/0 conn(0x7f22980b2210 legacy=0x7f229813f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:02.579 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 10 monclient: _finish_hunting 0 2023-12-11T20:16:02.579 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 1 monclient: found mon.noname-b 2023-12-11T20:16:02.580 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:02.580 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:02.580 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.578+0000 7f22977fe640 1 -- 172.21.15.133:0/240078718 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229813f720 con 0x7f22980a9b90 2023-12-11T20:16:02.580 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.580+0000 7f22977fe640 1 --2- 172.21.15.133:0/240078718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a9b90 0x7f229814c640 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2290009e50 tx=0x7f229002bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bd20778848a371d server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:02.583 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 1 -- 172.21.15.133:0/240078718 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229002c950 con 0x7f22980a9b90 2023-12-11T20:16:02.583 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.583 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:02.583 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: dump: 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: _finish_auth 0 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:02.584 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 1 -- 172.21.15.133:0/240078718 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229002cab0 con 0x7f22980a9b90 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f229d8cf640 5 monclient: authenticate success, global_id 4115 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f229d8cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f229d8cf640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 1 -- 172.21.15.133:0/240078718 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229002ce40 con 0x7f22980a9b90 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f229d8cf640 4 set_mon_vals no callback set 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.584+0000 7f2295ffb640 10 monclient: dump: 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.585 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 10 monclient: shutdown 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 1 -- 172.21.15.133:0/240078718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a9b90 msgr2=0x7f229814c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 1 --2- 172.21.15.133:0/240078718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a9b90 0x7f229814c640 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2290009e50 tx=0x7f229002bdb0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 1 -- 172.21.15.133:0/240078718 shutdown_connections 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 1 --2- 172.21.15.133:0/240078718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a9b90 0x7f229814c640 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 1 --2- 172.21.15.133:0/240078718 >> v2:172.21.15.133:3300/0 conn(0x7f22980a8930 0x7f2298145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.586 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 1 -- 172.21.15.133:0/240078718 >> 172.21.15.133:0/240078718 conn(0x7f22980b8380 msgr2=0x7f22980b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:02.587 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 1 -- 172.21.15.133:0/240078718 shutdown_connections 2023-12-11T20:16:02.587 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 1 -- 172.21.15.133:0/240078718 wait complete. 2023-12-11T20:16:02.590 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 10 monclient: build_initial_monmap 2023-12-11T20:16:02.590 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.585+0000 7f229d8cf640 10 monclient: monmap: 2023-12-11T20:16:02.590 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:02.590 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:02.590 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2023-12-11T20:16:02.590 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2023-12-11T20:16:02.590 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:02.590 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 1 librados: starting msgr at 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 1 librados: starting objecter 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 1 Processor -- start 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 1 -- start start 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 1 librados: setting wanted keys 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 1 librados: calling monclient init 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 10 monclient: init 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.586+0000 7f229d8cf640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:02.591 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a8930 0x7f22981dafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 10 monclient(hunting): picked mon.noname-c con 0x7f22980a8930 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22980a9b90 0x7f22981da660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 10 monclient(hunting): picked mon.noname-b con 0x7f22980a9b90 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22980b2210 0x7f2298140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 10 monclient(hunting): picked mon.noname-a con 0x7f22980b2210 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2298144cd0 con 0x7f22980b2210 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2298144ed0 con 0x7f22980a9b90 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2298136780 con 0x7f22980a8930 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f229d8cf640 10 monclient(hunting): authenticate will time out at 1455.935351s 2023-12-11T20:16:02.592 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f22967fc640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22980a9b90 0x7f22981da660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f22977fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22980b2210 0x7f2298140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.587+0000 7f22967fc640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22980a9b90 0x7f22981da660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:40306/0 (socket says 172.21.15.133:40306) 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22967fc640 1 -- 172.21.15.133:0/1717438359 learned_addr learned my addr 172.21.15.133:0/1717438359 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22967fc640 10 monclient(hunting): get_auth_request con 0x7f22980a9b90 auth_method 0 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22967fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22967fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22977fe640 10 monclient(hunting): get_auth_request con 0x7f22980b2210 auth_method 0 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22977fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22977fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.593 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.588+0000 7f22977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.589+0000 7f2296ffd640 1 --2- 172.21.15.133:0/1717438359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a8930 0x7f22981dafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.590+0000 7f2296ffd640 10 monclient(hunting): get_auth_request con 0x7f22980a8930 auth_method 0 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.590+0000 7f2296ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.590+0000 7f2296ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.590+0000 7f2296ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.590+0000 7f2296ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.590+0000 7f2296ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.590+0000 7f2296ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2023-12-11T20:16:02.594 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 1 -- 172.21.15.133:0/1717438359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a8930 msgr2=0x7f22981dafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.595 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 1 --2- 172.21.15.133:0/1717438359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a8930 0x7f22981dafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.595 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 1 -- 172.21.15.133:0/1717438359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22980b2210 msgr2=0x7f2298140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.595 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 1 --2- 172.21.15.133:0/1717438359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22980b2210 0x7f2298140170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.595 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 10 monclient: _finish_hunting 0 2023-12-11T20:16:02.595 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 1 monclient: found mon.noname-b 2023-12-11T20:16:02.595 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:02.595 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.595 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.591+0000 7f22967fc640 1 -- 172.21.15.133:0/1717438359 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2290009b30 con 0x7f22980a9b90 2023-12-11T20:16:02.596 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.593+0000 7f22967fc640 1 --2- 172.21.15.133:0/1717438359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22980a9b90 0x7f22981da660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f228800b3f0 tx=0x7f228800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.593+0000 7f227ffff640 10 client.?.objecter ms_handle_connect 0x7f22980a9b90 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.593+0000 7f227ffff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 1 -- 172.21.15.133:0/1717438359 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288011830 con 0x7f22980a9b90 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: dump: 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.597 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: _finish_auth 0 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 1 -- 172.21.15.133:0/1717438359 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2288011990 con 0x7f22980a9b90 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 1 -- 172.21.15.133:0/1717438359 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288011d40 con 0x7f22980a9b90 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f229d8cf640 5 monclient: authenticate success, global_id 4141 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:02.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f227ffff640 10 monclient: dump: 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.597+0000 7f229d8cf640 10 monclient: _renew_subs 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.597+0000 7f229d8cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.596+0000 7f22957fa640 4 set_mon_vals no callback set 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.597+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22981406b0 con 0x7f22980a9b90 2023-12-11T20:16:02.599 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.597+0000 7f229d8cf640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:02.600 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.597+0000 7f229d8cf640 10 monclient: _renew_subs 2023-12-11T20:16:02.600 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.597+0000 7f229d8cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.600 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.597+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22981358c0 con 0x7f22980a9b90 2023-12-11T20:16:02.600 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.597+0000 7f229d8cf640 1 librados: init done 2023-12-11T20:16:02.600 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.598+0000 7f229d8cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:02.600 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.598+0000 7f229d8cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:02.600 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.598+0000 7f229d8cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.600 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.598+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229804e840 con 0x7f22980a9b90 2023-12-11T20:16:02.601 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.599+0000 7f227ffff640 1 -- 172.21.15.133:0/1717438359 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2288004320 con 0x7f22980a9b90 2023-12-11T20:16:02.601 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.599+0000 7f227ffff640 1 -- 172.21.15.133:0/1717438359 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f228801a890 con 0x7f22980a9b90 2023-12-11T20:16:02.601 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.599+0000 7f227ffff640 10 client.4141.objecter ms_dispatch 0x7f2298000b90 osd_map(5..5 src has 1..5) v4 2023-12-11T20:16:02.601 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.599+0000 7f227ffff640 3 client.4141.objecter handle_osd_map got epochs [5,5] > 0 2023-12-11T20:16:02.601 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.599+0000 7f227ffff640 3 client.4141.objecter handle_osd_map decoding full epoch 5 2023-12-11T20:16:02.601 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.599+0000 7f227ffff640 20 client.4141.objecter dump_active .. 0 homeless 2023-12-11T20:16:02.626 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:02.627+0000 7f07fb213040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-11T20:16:02.682 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.684+0000 7fe472032040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-11T20:16:02.696 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.697+0000 7f227ffff640 1 -- 172.21.15.133:0/1717438359 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2288015020 con 0x7f22980a9b90 2023-12-11T20:16:02.696 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.697+0000 7f227ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:02.696 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.697+0000 7f227ffff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:02.728 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:02.729+0000 7f07fb213040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-11T20:16:02.740 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.742+0000 7fe472032040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-11T20:16:02.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.786+0000 7f229d8cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "73d8e78f-aa7e-47e7-9ce1-5571f6fc0759", "id": 3}] 2023-12-11T20:16:02.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.786+0000 7f229d8cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "73d8e78f-aa7e-47e7-9ce1-5571f6fc0759", "id": 3}] 2023-12-11T20:16:02.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.786+0000 7f229d8cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.786+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd new", "uuid": "73d8e78f-aa7e-47e7-9ce1-5571f6fc0759", "id": 3} v 0) v1 -- 0x7f229813fa20 con 0x7f22980a9b90 2023-12-11T20:16:02.824 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:02.825+0000 7f07fb213040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-11T20:16:02.854 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.856+0000 7f227ffff640 1 -- 172.21.15.133:0/1717438359 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "73d8e78f-aa7e-47e7-9ce1-5571f6fc0759", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f228801a4d0 con 0x7f22980a9b90 2023-12-11T20:16:02.854 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.856+0000 7f227ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "73d8e78f-aa7e-47e7-9ce1-5571f6fc0759", "id": 3}] 2023-12-11T20:16:02.854 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.856+0000 7f227ffff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:02.855 INFO:teuthology.orchestra.run.smithi133.stdout:3 2023-12-11T20:16:02.856 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.856+0000 7f229d8cf640 10 librados: watch_flush enter 2023-12-11T20:16:02.856 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.856+0000 7f229d8cf640 10 librados: watch_flush exit 2023-12-11T20:16:02.856 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.856+0000 7f229d8cf640 20 client.4141.objecter shutdown clearing up homeless session... 2023-12-11T20:16:02.856 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.856+0000 7f229d8cf640 10 client.4141.objecter successfully canceled tick 2023-12-11T20:16:02.856 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 10 monclient: shutdown 2023-12-11T20:16:02.856 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22980a9b90 msgr2=0x7f22981da660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 --2- 172.21.15.133:0/1717438359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22980a9b90 0x7f22981da660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f228800b3f0 tx=0x7f228800b8b0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 shutdown_connections 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 --2- 172.21.15.133:0/1717438359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22980b2210 0x7f2298140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 --2- 172.21.15.133:0/1717438359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22980a9b90 0x7f22981da660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 --2- 172.21.15.133:0/1717438359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22980a8930 0x7f22981dafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 >> 172.21.15.133:0/1717438359 conn(0x7f22980b8380 msgr2=0x7f229813ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 shutdown_connections 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 -- 172.21.15.133:0/1717438359 wait complete. 2023-12-11T20:16:02.857 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.857+0000 7f229d8cf640 1 librados: shutdown 2023-12-11T20:16:02.865 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new f9926b1b-1b80-43ad-beaa-cd6645b9b72c 4 2023-12-11T20:16:02.886 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:02.888+0000 7fe472032040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.967+0000 7f19cf365640 10 monclient: get_monmap_and_config 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.967+0000 7f19cf365640 10 monclient: build_initial_monmap 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.967+0000 7f19cf365640 10 monclient: monmap: 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:16:02.967998+0000 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:16:02.967998+0000 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.966 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:02.967 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:02.967 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:02.967 INFO:teuthology.orchestra.run.smithi133.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:02.967 INFO:teuthology.orchestra.run.smithi133.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:02.967 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.967 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.967+0000 7f19cf365640 1 Processor -- start 2023-12-11T20:16:02.967 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.967+0000 7f19cf365640 1 -- start start 2023-12-11T20:16:02.967 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.967+0000 7f19cf365640 10 monclient: init 2023-12-11T20:16:02.970 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:02.970 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-12-11T20:16:02.970 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19c80a8930 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:02.970 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19c80b21a0 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c813f7c0 0x7f19c80a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 10 monclient(hunting): picked mon.noname-b con 0x7f19c813f7c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c80aa980 con 0x7f19c80b21a0 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c80aabf0 con 0x7f19c80a8930 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c80aaed0 con 0x7f19c813f7c0 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.968+0000 7f19cf365640 10 monclient(hunting): authenticate will time out at 1456.316792s 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.969+0000 7f19cd902640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c813f7c0 0x7f19c80a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.969+0000 7f19cd101640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f19c80a8930 0x7f19c81424b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:45706/0 (socket says 172.21.15.133:45706) 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.969+0000 7f19cd101640 1 -- 172.21.15.133:0/851338501 learned_addr learned my addr 172.21.15.133:0/851338501 (peer_addr_for_me v1:172.21.15.133:0/0) 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.969+0000 7f19cc900640 1 --1- 172.21.15.133:0/851338501 >> v1:172.21.15.86:6789/0 conn(0x7f19c80b21a0 0x7f19c813f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.133:37582/0 (socket says 172.21.15.133:37582) 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.969+0000 7f19cd902640 10 monclient(hunting): get_auth_request con 0x7f19c813f7c0 auth_method 0 2023-12-11T20:16:02.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.969+0000 7f19cd902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.969+0000 7f19cd902640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.969+0000 7f19cd902640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.970+0000 7f19cd902640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.970+0000 7f19cd902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.970+0000 7f19cd902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 1 -- 172.21.15.133:0/851338501 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338710165 0 0) 0x7f19c80aa980 con 0x7f19c80b21a0 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 10 monclient(hunting): _negotiate 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 10 monclient(hunting): my global_id is 4134 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 1 -- 172.21.15.133:0/851338501 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b4003520 con 0x7f19c80b21a0 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 1 -- 172.21.15.133:0/851338501 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773841480 0 0) 0x7f19c80aabf0 con 0x7f19c80a8930 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 10 monclient(hunting): _negotiate 2023-12-11T20:16:02.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 10 monclient(hunting): my global_id is 4144 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19bffff640 1 -- 172.21.15.133:0/851338501 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c80aa980 con 0x7f19c80a8930 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19cd902640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19cd902640 1 -- 172.21.15.133:0/851338501 >> v1:172.21.15.133:6789/0 conn(0x7f19c80a8930 legacy=0x7f19c81424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19cd902640 1 -- 172.21.15.133:0/851338501 >> v1:172.21.15.86:6789/0 conn(0x7f19c80b21a0 legacy=0x7f19c813f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19cd902640 10 monclient: _finish_hunting 0 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19cd902640 1 monclient: found mon.noname-b 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19cd902640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19cd902640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:02.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.971+0000 7f19cd902640 1 -- 172.21.15.133:0/851338501 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8141f90 con 0x7f19c813f7c0 2023-12-11T20:16:02.974 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.973+0000 7f19cd902640 1 --2- 172.21.15.133:0/851338501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c813f7c0 0x7f19c80a8d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009df0 tx=0x7f19c002be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2cde5dbac4dd53 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:02.976 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 1 -- 172.21.15.133:0/851338501 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c002cc20 con 0x7f19c813f7c0 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: dump: 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.977 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: _finish_auth 0 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 1 -- 172.21.15.133:0/851338501 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c002cd80 con 0x7f19c813f7c0 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19cf365640 5 monclient: authenticate success, global_id 4121 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19cf365640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19cf365640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19cf365640 4 set_mon_vals no callback set 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 1 -- 172.21.15.133:0/851338501 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0034660 con 0x7f19c813f7c0 2023-12-11T20:16:02.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.978+0000 7f19bffff640 10 monclient: dump: 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.979 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 10 monclient: shutdown 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 1 -- 172.21.15.133:0/851338501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c813f7c0 msgr2=0x7f19c80a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 1 --2- 172.21.15.133:0/851338501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c813f7c0 0x7f19c80a8d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009df0 tx=0x7f19c002be70 comp rx=0 tx=0).stop 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 1 -- 172.21.15.133:0/851338501 shutdown_connections 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 1 --2- 172.21.15.133:0/851338501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c813f7c0 0x7f19c80a8d10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 1 -- 172.21.15.133:0/851338501 >> 172.21.15.133:0/851338501 conn(0x7f19c80b82f0 msgr2=0x7f19c80b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 1 -- 172.21.15.133:0/851338501 shutdown_connections 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 1 -- 172.21.15.133:0/851338501 wait complete. 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 10 monclient: build_initial_monmap 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.979+0000 7f19cf365640 10 monclient: monmap: 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:02.984 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 librados: starting msgr at 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 librados: starting objecter 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 Processor -- start 2023-12-11T20:16:02.985 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 -- start start 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 librados: setting wanted keys 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 librados: calling monclient init 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient: init 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c80a8930 0x7f19c80b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient(hunting): picked mon.noname-c con 0x7f19c80a8930 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19c80b21a0 0x7f19c8136b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient(hunting): picked mon.noname-b con 0x7f19c80b21a0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c813f7c0 0x7f19c81da180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient(hunting): picked mon.noname-a con 0x7f19c813f7c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c8145bb0 con 0x7f19c813f7c0 2023-12-11T20:16:02.986 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c8138140 con 0x7f19c80b21a0 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c8145690 con 0x7f19c80a8930 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.980+0000 7f19cf365640 10 monclient(hunting): authenticate will time out at 1456.329108s 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cc900640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19c80b21a0 0x7f19c8136b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd902640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c813f7c0 0x7f19c81da180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd101640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c80a8930 0x7f19c80b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cc900640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19c80b21a0 0x7f19c8136b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:40316/0 (socket says 172.21.15.133:40316) 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cc900640 1 -- 172.21.15.133:0/1605501794 learned_addr learned my addr 172.21.15.133:0/1605501794 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cc900640 10 monclient(hunting): get_auth_request con 0x7f19c80b21a0 auth_method 0 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cc900640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cc900640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cc900640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.987 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd101640 10 monclient(hunting): get_auth_request con 0x7f19c80a8930 auth_method 0 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd101640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd101640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd101640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd902640 10 monclient(hunting): get_auth_request con 0x7f19c813f7c0 auth_method 0 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd902640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.981+0000 7f19cd902640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cc900640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cc900640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cc900640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cd101640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.988 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cd101640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cd101640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cd902640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cd902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.982+0000 7f19cd902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.984+0000 7f19cc900640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.984+0000 7f19cc900640 1 -- 172.21.15.133:0/1605501794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c80a8930 msgr2=0x7f19c80b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.984+0000 7f19cc900640 1 --2- 172.21.15.133:0/1605501794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c80a8930 0x7f19c80b7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.984+0000 7f19cc900640 1 -- 172.21.15.133:0/1605501794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c813f7c0 msgr2=0x7f19c81da180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.984+0000 7f19cc900640 1 --2- 172.21.15.133:0/1605501794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c813f7c0 0x7f19c81da180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.985+0000 7f19cc900640 10 monclient: _finish_hunting 0 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.985+0000 7f19cc900640 1 monclient: found mon.noname-b 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.985+0000 7f19cc900640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.985+0000 7f19cc900640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.989 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.985+0000 7f19cc900640 1 -- 172.21.15.133:0/1605501794 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c0009a70 con 0x7f19c80b21a0 2023-12-11T20:16:02.990 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:02.986+0000 7f07fb213040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.987+0000 7f19cc900640 1 --2- 172.21.15.133:0/1605501794 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19c80b21a0 0x7f19c8136b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f19c400c450 tx=0x7f19c400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.987+0000 7f19bdffb640 10 client.?.objecter ms_handle_connect 0x7f19c80b21a0 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.987+0000 7f19bdffb640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 1 -- 172.21.15.133:0/1605501794 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4007480 con 0x7f19c80b21a0 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: dump: 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.992 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: _finish_auth 0 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: _check_auth_tickets 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 1 -- 172.21.15.133:0/1605501794 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c40075e0 con 0x7f19c80b21a0 2023-12-11T20:16:02.993 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 1 -- 172.21.15.133:0/1605501794 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4004a20 con 0x7f19c80b21a0 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19bdffb640 10 monclient: dump: 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:02.994 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 5 monclient: authenticate success, global_id 4147 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19beffd640 4 set_mon_vals no callback set 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 10 monclient: _renew_subs 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c8145200 con 0x7f19c80b21a0 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 10 monclient: _renew_subs 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c81da8a0 con 0x7f19c80b21a0 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.990+0000 7f19cf365640 1 librados: init done 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.992+0000 7f19cf365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:02.995 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.992+0000 7f19cf365640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:02.996 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.992+0000 7f19cf365640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:02.996 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.992+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c804e840 con 0x7f19c80b21a0 2023-12-11T20:16:02.996 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.993+0000 7f19bdffb640 1 -- 172.21.15.133:0/1605501794 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f19c4012440 con 0x7f19c80b21a0 2023-12-11T20:16:02.996 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.993+0000 7f19bdffb640 1 -- 172.21.15.133:0/1605501794 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f19c401c950 con 0x7f19c80b21a0 2023-12-11T20:16:02.996 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.993+0000 7f19bdffb640 10 client.4147.objecter ms_dispatch 0x7f19c8000b90 osd_map(6..6 src has 1..6) v4 2023-12-11T20:16:02.996 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.993+0000 7f19bdffb640 3 client.4147.objecter handle_osd_map got epochs [6,6] > 0 2023-12-11T20:16:02.996 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.993+0000 7f19bdffb640 3 client.4147.objecter handle_osd_map decoding full epoch 6 2023-12-11T20:16:02.996 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:02.993+0000 7f19bdffb640 20 client.4147.objecter dump_active .. 0 homeless 2023-12-11T20:16:03.028 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:03.029+0000 7f07fb213040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-11T20:16:03.089 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.091+0000 7f19bdffb640 1 -- 172.21.15.133:0/1605501794 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f19c4018070 con 0x7f19c80b21a0 2023-12-11T20:16:03.090 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.091+0000 7f19bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:03.090 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.091+0000 7f19bdffb640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:03.166 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:03.166+0000 7f07fb213040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-11T20:16:03.178 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.179+0000 7f19cf365640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f9926b1b-1b80-43ad-beaa-cd6645b9b72c", "id": 4}] 2023-12-11T20:16:03.178 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.180+0000 7f19cf365640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f9926b1b-1b80-43ad-beaa-cd6645b9b72c", "id": 4}] 2023-12-11T20:16:03.178 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.180+0000 7f19cf365640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:03.178 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.180+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f9926b1b-1b80-43ad-beaa-cd6645b9b72c", "id": 4} v 0) v1 -- 0x7f19c8145850 con 0x7f19c80b21a0 2023-12-11T20:16:03.240 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.242+0000 7f19bdffb640 1 -- 172.21.15.133:0/1605501794 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f9926b1b-1b80-43ad-beaa-cd6645b9b72c", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f19c401cce0 con 0x7f19c80b21a0 2023-12-11T20:16:03.240 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.242+0000 7f19bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f9926b1b-1b80-43ad-beaa-cd6645b9b72c", "id": 4}] 2023-12-11T20:16:03.240 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.242+0000 7f19bdffb640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:03.241 INFO:teuthology.orchestra.run.smithi133.stdout:4 2023-12-11T20:16:03.242 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 10 librados: watch_flush enter 2023-12-11T20:16:03.242 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 10 librados: watch_flush exit 2023-12-11T20:16:03.242 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 20 client.4147.objecter shutdown clearing up homeless session... 2023-12-11T20:16:03.242 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 10 client.4147.objecter successfully canceled tick 2023-12-11T20:16:03.242 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 10 monclient: shutdown 2023-12-11T20:16:03.242 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:03.242 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19c80b21a0 msgr2=0x7f19c8136b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 --2- 172.21.15.133:0/1605501794 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19c80b21a0 0x7f19c8136b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f19c400c450 tx=0x7f19c400c910 comp rx=0 tx=0).stop 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 shutdown_connections 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 --2- 172.21.15.133:0/1605501794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19c813f7c0 0x7f19c81da180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 --2- 172.21.15.133:0/1605501794 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19c80b21a0 0x7f19c8136b50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 --2- 172.21.15.133:0/1605501794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19c80a8930 0x7f19c80b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 >> 172.21.15.133:0/1605501794 conn(0x7f19c80b82f0 msgr2=0x7f19c80b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 shutdown_connections 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 -- 172.21.15.133:0/1605501794 wait complete. 2023-12-11T20:16:03.243 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.243+0000 7f19cf365640 1 librados: shutdown 2023-12-11T20:16:03.252 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 15e64f16-9d47-453f-a088-8c124c32c5d5 5 2023-12-11T20:16:03.394 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.396+0000 7f6bcdef7640 10 monclient: get_monmap_and_config 2023-12-11T20:16:03.394 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.396+0000 7f6bcdef7640 10 monclient: build_initial_monmap 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.396+0000 7f6bcdef7640 10 monclient: monmap: 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:16:03.397003+0000 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:16:03.397003+0000 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:03.395 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.396+0000 7f6bcdef7640 1 Processor -- start 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.396+0000 7f6bcdef7640 1 -- start start 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.396+0000 7f6bcdef7640 10 monclient: init 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 0x7f6bc80a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient(hunting): picked mon.noname-b con 0x7f6bc80a8930 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6bc80b21a0 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f6bc813f0b0 0x7f6bc813f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient(hunting): picked mon.noname-c con 0x7f6bc813f0b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc813fa60 con 0x7f6bc80b21a0 2023-12-11T20:16:03.397 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.398 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc80aa980 con 0x7f6bc813f0b0 2023-12-11T20:16:03.398 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.398 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc80aabf0 con 0x7f6bc80a8930 2023-12-11T20:16:03.398 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:03.398 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.397+0000 7f6bcdef7640 10 monclient(hunting): authenticate will time out at 1456.745846s 2023-12-11T20:16:03.398 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.400+0000 7f6bc7fff640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f6bc813f0b0 0x7f6bc813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.399 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.401+0000 7f6bc6ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f6bc80b21a0 0x7f6bc81424b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:45716/0 (socket says 172.21.15.133:45716) 2023-12-11T20:16:03.399 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.401+0000 7f6bc6ffd640 1 -- 172.21.15.133:0/1352706915 learned_addr learned my addr 172.21.15.133:0/1352706915 (peer_addr_for_me v1:172.21.15.133:0/0) 2023-12-11T20:16:03.399 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.401+0000 7f6bc7fff640 10 monclient(hunting): get_auth_request con 0x7f6bc813f0b0 auth_method 0 2023-12-11T20:16:03.399 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.401+0000 7f6bc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.399 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.401+0000 7f6bc7fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.399 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.401+0000 7f6bc7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.400 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.402+0000 7f6bc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.400 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.402+0000 7f6bc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.400 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.402+0000 7f6bc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.403 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2023-12-11T20:16:03.403 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 1 -- 172.21.15.133:0/1352706915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 msgr2=0x7f6bc80a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:03.403 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 1 --2- 172.21.15.133:0/1352706915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 0x7f6bc80a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.404 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 1 -- 172.21.15.133:0/1352706915 >> v1:172.21.15.133:6789/0 conn(0x7f6bc80b21a0 legacy=0x7f6bc81424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.404 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:03.404 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 1 monclient: found mon.noname-c 2023-12-11T20:16:03.404 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:03.404 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:03.404 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.405+0000 7f6bc7fff640 1 -- 172.21.15.133:0/1352706915 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc80aaed0 con 0x7f6bc813f0b0 2023-12-11T20:16:03.406 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.408+0000 7f6bc7fff640 1 --2- 172.21.15.133:0/1352706915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc813f520 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009d90 tx=0x7f6bc002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb53d0b60a137f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:03.410 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 1 -- 172.21.15.133:0/1352706915 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc002cbc0 con 0x7f6bc813f0b0 2023-12-11T20:16:03.410 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:03.410 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:03.410 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: dump: 2023-12-11T20:16:03.410 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:03.411 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:03.412 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 1 -- 172.21.15.133:0/1352706915 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc002cd20 con 0x7f6bc813f0b0 2023-12-11T20:16:03.412 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 5 monclient: authenticate success, global_id 4150 2023-12-11T20:16:03.412 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:03.412 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 4 set_mon_vals no callback set 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 1 -- 172.21.15.133:0/1352706915 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0034660 con 0x7f6bc813f0b0 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bc67fc640 10 monclient: dump: 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:03.413 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:03.414 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:03.414 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.417 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 10 monclient: shutdown 2023-12-11T20:16:03.417 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:03.417 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1352706915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 msgr2=0x7f6bc813f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.417 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 1 --2- 172.21.15.133:0/1352706915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc813f520 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009d90 tx=0x7f6bc002be10 comp rx=0 tx=0).stop 2023-12-11T20:16:03.417 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1352706915 shutdown_connections 2023-12-11T20:16:03.417 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 1 --2- 172.21.15.133:0/1352706915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc813f520 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.417 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 1 --2- 172.21.15.133:0/1352706915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 0x7f6bc80a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1352706915 >> 172.21.15.133:0/1352706915 conn(0x7f6bc80b82f0 msgr2=0x7f6bc80b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1352706915 shutdown_connections 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.412+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1352706915 wait complete. 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.413+0000 7f6bcdef7640 10 monclient: build_initial_monmap 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.413+0000 7f6bcdef7640 10 monclient: monmap: 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:03.418 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 librados: starting msgr at 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 librados: starting objecter 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 Processor -- start 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 -- start start 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 librados: setting wanted keys 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 librados: calling monclient init 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient: init 2023-12-11T20:16:03.419 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 0x7f6bc8136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient(hunting): picked mon.noname-c con 0x7f6bc80a8930 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc80b21a0 0x7f6bc8144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient(hunting): picked mon.noname-a con 0x7f6bc80b21a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc8145a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient(hunting): picked mon.noname-b con 0x7f6bc813f0b0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc8145260 con 0x7f6bc80b21a0 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc8145590 con 0x7f6bc813f0b0 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.420 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc8138a80 con 0x7f6bc80a8930 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.414+0000 7f6bcdef7640 10 monclient(hunting): authenticate will time out at 1456.763208s 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc7fff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc8145a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc80b21a0 0x7f6bc8144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 0x7f6bc8136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc7fff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc8145a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:40332/0 (socket says 172.21.15.133:40332) 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc7fff640 1 -- 172.21.15.133:0/1692155287 learned_addr learned my addr 172.21.15.133:0/1692155287 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc7fff640 10 monclient(hunting): get_auth_request con 0x7f6bc813f0b0 auth_method 0 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc7fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc77fe640 10 monclient(hunting): get_auth_request con 0x7f6bc80a8930 auth_method 0 2023-12-11T20:16:03.421 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc6ffd640 10 monclient(hunting): get_auth_request con 0x7f6bc80b21a0 auth_method 0 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.415+0000 7f6bc6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.422 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.416+0000 7f6bc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 1 -- 172.21.15.133:0/1692155287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 msgr2=0x7f6bc8136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 1 --2- 172.21.15.133:0/1692155287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 0x7f6bc8136fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 1 -- 172.21.15.133:0/1692155287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc80b21a0 msgr2=0x7f6bc8144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 1 --2- 172.21.15.133:0/1692155287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc80b21a0 0x7f6bc8144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 1 monclient: found mon.noname-b 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:03.423 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.418+0000 7f6bc7fff640 1 -- 172.21.15.133:0/1692155287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc0009a70 con 0x7f6bc813f0b0 2023-12-11T20:16:03.425 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.421+0000 7f6bc7fff640 1 --2- 172.21.15.133:0/1692155287 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc8145a20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009eb0 tx=0x7f6bc0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:03.425 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.421+0000 7f6baffff640 10 client.?.objecter ms_handle_connect 0x7f6bc813f0b0 2023-12-11T20:16:03.425 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.421+0000 7f6baffff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:03.425 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 1 -- 172.21.15.133:0/1692155287 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc003f420 con 0x7f6bc813f0b0 2023-12-11T20:16:03.425 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:03.425 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:03.425 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: dump: 2023-12-11T20:16:03.425 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: _finish_auth 0 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 5 monclient: authenticate success, global_id 4156 2023-12-11T20:16:03.426 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 1 -- 172.21.15.133:0/1692155287 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc002ce90 con 0x7f6bc813f0b0 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 1 -- 172.21.15.133:0/1692155287 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc003f780 con 0x7f6bc813f0b0 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 10 monclient: _renew_subs 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc8138d60 con 0x7f6bc813f0b0 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bc5ffb640 4 set_mon_vals no callback set 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6baffff640 10 monclient: dump: 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:03.427 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 10 monclient: _renew_subs 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc8146140 con 0x7f6bc813f0b0 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.424+0000 7f6bcdef7640 1 librados: init done 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.425+0000 7f6bcdef7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.425+0000 7f6bcdef7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.425+0000 7f6bcdef7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:03.428 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.425+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc804e840 con 0x7f6bc813f0b0 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 1 -- 172.21.15.133:0/1692155287 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6bc003f8e0 con 0x7f6bc813f0b0 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 1 -- 172.21.15.133:0/1692155287 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f6bc0052e00 con 0x7f6bc813f0b0 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 10 client.4156.objecter ms_dispatch 0x7f6bc8000b90 osd_map(7..7 src has 1..7) v4 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 3 client.4156.objecter handle_osd_map got epochs [7,7] > 0 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 3 client.4156.objecter handle_osd_map decoding full epoch 7 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 20 client.4156.objecter dump_active .. 0 homeless 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 1 -- 172.21.15.133:0/1692155287 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f6bc004d060 con 0x7f6bc813f0b0 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:03.526 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.527+0000 7f6baffff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:03.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.600+0000 7f6bcdef7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "15e64f16-9d47-453f-a088-8c124c32c5d5", "id": 5}] 2023-12-11T20:16:03.598 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.600+0000 7f6bcdef7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "15e64f16-9d47-453f-a088-8c124c32c5d5", "id": 5}] 2023-12-11T20:16:03.599 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.600+0000 7f6bcdef7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:03.599 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.600+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd new", "uuid": "15e64f16-9d47-453f-a088-8c124c32c5d5", "id": 5} v 0) v1 -- 0x7f6bc81456d0 con 0x7f6bc813f0b0 2023-12-11T20:16:03.668 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.669+0000 7f6baffff640 1 -- 172.21.15.133:0/1692155287 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "15e64f16-9d47-453f-a088-8c124c32c5d5", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6bc00560b0 con 0x7f6bc813f0b0 2023-12-11T20:16:03.668 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.669+0000 7f6baffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "15e64f16-9d47-453f-a088-8c124c32c5d5", "id": 5}] 2023-12-11T20:16:03.668 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.669+0000 7f6baffff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:03.669 INFO:teuthology.orchestra.run.smithi133.stdout:5 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 10 librados: watch_flush enter 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 10 librados: watch_flush exit 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 20 client.4156.objecter shutdown clearing up homeless session... 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 10 client.4156.objecter successfully canceled tick 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 10 monclient: shutdown 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 msgr2=0x7f6bc8145a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 1 --2- 172.21.15.133:0/1692155287 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc8145a20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009eb0 tx=0x7f6bc0004a40 comp rx=0 tx=0).stop 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 shutdown_connections 2023-12-11T20:16:03.670 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.670+0000 7f6bcdef7640 1 --2- 172.21.15.133:0/1692155287 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6bc813f0b0 0x7f6bc8145a20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.671 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.671+0000 7f6bcdef7640 1 --2- 172.21.15.133:0/1692155287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bc80b21a0 0x7f6bc8144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.671 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.671+0000 7f6bcdef7640 1 --2- 172.21.15.133:0/1692155287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bc80a8930 0x7f6bc8136fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.671 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.671+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 >> 172.21.15.133:0/1692155287 conn(0x7f6bc80b82f0 msgr2=0x7f6bc80b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:03.671 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.671+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 shutdown_connections 2023-12-11T20:16:03.671 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.671+0000 7f6bcdef7640 1 -- 172.21.15.133:0/1692155287 wait complete. 2023-12-11T20:16:03.671 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.671+0000 7f6bcdef7640 1 librados: shutdown 2023-12-11T20:16:03.680 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new b44f36ee-4393-49e2-a7cf-8f4cec897bad 6 2023-12-11T20:16:03.732 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:03.733+0000 7fe472032040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-11T20:16:03.779 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.781+0000 7f5f69d61640 10 monclient: get_monmap_and_config 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.781+0000 7f5f69d61640 10 monclient: build_initial_monmap 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.781+0000 7f5f69d61640 10 monclient: monmap: 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:16:03.782382+0000 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:16:03.782382+0000 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:03.780 INFO:teuthology.orchestra.run.smithi133.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:03.781 INFO:teuthology.orchestra.run.smithi133.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:03.781 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.783 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 1 Processor -- start 2023-12-11T20:16:03.783 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 1 -- start start 2023-12-11T20:16:03.783 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient: init 2023-12-11T20:16:03.783 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:03.783 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-12-11T20:16:03.783 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f640618a0 0x7f5f640716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.784 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient(hunting): picked mon.noname-a con 0x7f5f640618a0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:03.784 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f6406c120 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:03.784 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5f6413efb0 0x7f5f6413f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.784 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6413efb0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:03.784 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64070d80 con 0x7f5f6406c120 2023-12-11T20:16:03.784 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f64061c80 con 0x7f5f640618a0 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f64071c30 con 0x7f5f6413efb0 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.782+0000 7f5f69d61640 10 monclient(hunting): authenticate will time out at 1457.131208s 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.783+0000 7f5f63fff640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5f6413efb0 0x7f5f6413f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.783+0000 7f5f637fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f640618a0 0x7f5f640716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f62ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5f6406c120 0x7f5f6406cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.133:37596/0 (socket says 172.21.15.133:37596) 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f62ffd640 1 -- 172.21.15.133:0/1700616571 learned_addr learned my addr 172.21.15.133:0/1700616571 (peer_addr_for_me v1:172.21.15.133:0/0) 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f63fff640 10 monclient(hunting): get_auth_request con 0x7f5f6413efb0 auth_method 0 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f63fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f63fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f637fe640 10 monclient(hunting): get_auth_request con 0x7f5f640618a0 auth_method 0 2023-12-11T20:16:03.785 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.786 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f637fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.786 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f637fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.786 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.786 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.786 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.784+0000 7f5f63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.786 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.785+0000 7f5f637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.786 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.785+0000 7f5f637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.786 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.785+0000 7f5f637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 10 monclient(hunting): handle_auth_done global_id 4159 payload 274 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 1 -- 172.21.15.133:0/1700616571 >> v2:172.21.15.86:3300/0 conn(0x7f5f640618a0 msgr2=0x7f5f640716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 1 --2- 172.21.15.133:0/1700616571 >> v2:172.21.15.86:3300/0 conn(0x7f5f640618a0 0x7f5f640716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 1 -- 172.21.15.133:0/1700616571 >> v1:172.21.15.86:6789/0 conn(0x7f5f6406c120 legacy=0x7f5f6406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 1 monclient: found mon.noname-c 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:03.787 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.786+0000 7f5f63fff640 1 -- 172.21.15.133:0/1700616571 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f64070ff0 con 0x7f5f6413efb0 2023-12-11T20:16:03.788 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.788+0000 7f5f63fff640 1 --2- 172.21.15.133:0/1700616571 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f6413efb0 0x7f5f6413f400 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f64070c40 tx=0x7f5f5c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0837c2d8e909406 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:03.790 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 1 -- 172.21.15.133:0/1700616571 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c02cc80 con 0x7f5f6413efb0 2023-12-11T20:16:03.790 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:03.790 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:03.790 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: dump: 2023-12-11T20:16:03.790 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:03.790 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.790 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 1 -- 172.21.15.133:0/1700616571 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c02cde0 con 0x7f5f6413efb0 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f69d61640 5 monclient: authenticate success, global_id 4159 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f69d61640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:03.791 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 1 -- 172.21.15.133:0/1700616571 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c034660 con 0x7f5f6413efb0 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.791+0000 7f5f627fc640 10 monclient: dump: 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.792 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:03.793 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:03.793 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:03.793 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.793 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:03.793 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 4 set_mon_vals no callback set 2023-12-11T20:16:03.797 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 10 monclient: shutdown 2023-12-11T20:16:03.797 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:03.797 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 1 -- 172.21.15.133:0/1700616571 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f6413efb0 msgr2=0x7f5f6413f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.797 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 1 --2- 172.21.15.133:0/1700616571 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f6413efb0 0x7f5f6413f400 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f64070c40 tx=0x7f5f5c02bed0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.797 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 1 -- 172.21.15.133:0/1700616571 shutdown_connections 2023-12-11T20:16:03.797 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 1 --2- 172.21.15.133:0/1700616571 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f6413efb0 0x7f5f6413f400 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.797 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 1 --2- 172.21.15.133:0/1700616571 >> v2:172.21.15.86:3300/0 conn(0x7f5f640618a0 0x7f5f640716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.797 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 1 -- 172.21.15.133:0/1700616571 >> 172.21.15.133:0/1700616571 conn(0x7f5f640b28c0 msgr2=0x7f5f64060f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 1 -- 172.21.15.133:0/1700616571 shutdown_connections 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.792+0000 7f5f69d61640 1 -- 172.21.15.133:0/1700616571 wait complete. 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.793+0000 7f5f69d61640 10 monclient: build_initial_monmap 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.793+0000 7f5f69d61640 10 monclient: monmap: 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:03.798 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.793+0000 7f5f69d61640 1 librados: starting msgr at 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.793+0000 7f5f69d61640 1 librados: starting objecter 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.793+0000 7f5f69d61640 1 Processor -- start 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 -- start start 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 librados: setting wanted keys 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 librados: calling monclient init 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient: init 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f640618a0 0x7f5f641367f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient(hunting): picked mon.noname-b con 0x7f5f640618a0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f6406c120 0x7f5f64133820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.799 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6406c120 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f6413efb0 0x7f5f64133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6413efb0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f64138110 con 0x7f5f6413efb0 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f64137080 con 0x7f5f640618a0 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f64136d30 con 0x7f5f6406c120 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.794+0000 7f5f69d61640 10 monclient(hunting): authenticate will time out at 1457.142895s 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f640618a0 0x7f5f641367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f640618a0 0x7f5f641367f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:40340/0 (socket says 172.21.15.133:40340) 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 1 -- 172.21.15.133:0/1035501781 learned_addr learned my addr 172.21.15.133:0/1035501781 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 10 monclient(hunting): get_auth_request con 0x7f5f640618a0 auth_method 0 2023-12-11T20:16:03.800 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f63fff640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f6413efb0 0x7f5f64133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f62ffd640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f6406c120 0x7f5f64133820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f63fff640 10 monclient(hunting): get_auth_request con 0x7f5f6413efb0 auth_method 0 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f63fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f63fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f62ffd640 10 monclient(hunting): get_auth_request con 0x7f5f6406c120 auth_method 0 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f62ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f62ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.801 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.795+0000 7f5f637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.796+0000 7f5f63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.796+0000 7f5f63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.796+0000 7f5f63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.796+0000 7f5f62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.796+0000 7f5f62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.796+0000 7f5f62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 1 -- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f6406c120 msgr2=0x7f5f64133820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f6406c120 0x7f5f64133820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 1 -- 172.21.15.133:0/1035501781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f640618a0 msgr2=0x7f5f641367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f640618a0 0x7f5f641367f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 1 monclient: found mon.noname-a 2023-12-11T20:16:03.802 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:03.803 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:03.803 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.798+0000 7f5f63fff640 1 -- 172.21.15.133:0/1035501781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c009a70 con 0x7f5f6413efb0 2023-12-11T20:16:03.804 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.800+0000 7f5f63fff640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f6413efb0 0x7f5f64133d60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c033000 tx=0x7f5f5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:03.804 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.800+0000 7f5f4bfff640 10 client.?.objecter ms_handle_connect 0x7f5f6413efb0 2023-12-11T20:16:03.804 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.800+0000 7f5f4bfff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:03.804 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 1 -- 172.21.15.133:0/1035501781 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c03e610 con 0x7f5f6413efb0 2023-12-11T20:16:03.804 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:03.804 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: dump: 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: _finish_auth 0 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:03.805 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 1 -- 172.21.15.133:0/1035501781 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c03e770 con 0x7f5f6413efb0 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 1 -- 172.21.15.133:0/1035501781 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c03eb20 con 0x7f5f6413efb0 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 5 monclient: authenticate success, global_id 4149 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 10 monclient: _renew_subs 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f641342a0 con 0x7f5f6413efb0 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f61ffb640 4 set_mon_vals no callback set 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f4bfff640 10 monclient: dump: 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.806 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 10 monclient: _renew_subs 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f641386a0 con 0x7f5f6413efb0 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.804+0000 7f5f69d61640 1 librados: init done 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.805+0000 7f5f69d61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.805+0000 7f5f69d61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.805+0000 7f5f69d61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:03.807 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.805+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f641d6e10 con 0x7f5f6413efb0 2023-12-11T20:16:03.808 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.807+0000 7f5f4bfff640 1 -- 172.21.15.133:0/1035501781 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5f5c03b050 con 0x7f5f6413efb0 2023-12-11T20:16:03.808 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.807+0000 7f5f4bfff640 1 -- 172.21.15.133:0/1035501781 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f5f5c053b80 con 0x7f5f6413efb0 2023-12-11T20:16:03.808 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.807+0000 7f5f4bfff640 10 client.4149.objecter ms_dispatch 0x7f5f64000b90 osd_map(8..8 src has 1..8) v4 2023-12-11T20:16:03.808 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.807+0000 7f5f4bfff640 3 client.4149.objecter handle_osd_map got epochs [8,8] > 0 2023-12-11T20:16:03.808 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.807+0000 7f5f4bfff640 3 client.4149.objecter handle_osd_map decoding full epoch 8 2023-12-11T20:16:03.808 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.808+0000 7f5f4bfff640 20 client.4149.objecter dump_active .. 0 homeless 2023-12-11T20:16:03.905 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.907+0000 7f5f4bfff640 1 -- 172.21.15.133:0/1035501781 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f5f641d6e10 con 0x7f5f6413efb0 2023-12-11T20:16:03.905 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.907+0000 7f5f4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:03.905 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.907+0000 7f5f4bfff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:03.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.979+0000 7f5f69d61640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b44f36ee-4393-49e2-a7cf-8f4cec897bad", "id": 6}] 2023-12-11T20:16:03.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.979+0000 7f5f69d61640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b44f36ee-4393-49e2-a7cf-8f4cec897bad", "id": 6}] 2023-12-11T20:16:03.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.979+0000 7f5f69d61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:03.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:03.979+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b44f36ee-4393-49e2-a7cf-8f4cec897bad", "id": 6} v 0) v1 -- 0x7f5f6404e840 con 0x7f5f6413efb0 2023-12-11T20:16:04.009 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.011+0000 7f5f4bfff640 1 -- 172.21.15.133:0/1035501781 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b44f36ee-4393-49e2-a7cf-8f4cec897bad", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5f5c04e060 con 0x7f5f6413efb0 2023-12-11T20:16:04.010 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.011+0000 7f5f4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b44f36ee-4393-49e2-a7cf-8f4cec897bad", "id": 6}] 2023-12-11T20:16:04.010 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.011+0000 7f5f4bfff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:04.010 INFO:teuthology.orchestra.run.smithi133.stdout:6 2023-12-11T20:16:04.011 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 10 librados: watch_flush enter 2023-12-11T20:16:04.011 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 10 librados: watch_flush exit 2023-12-11T20:16:04.011 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 20 client.4149.objecter shutdown clearing up homeless session... 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 10 client.4149.objecter successfully canceled tick 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 10 monclient: shutdown 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f6413efb0 msgr2=0x7f5f64133d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f6413efb0 0x7f5f64133d60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c033000 tx=0x7f5f5c004a40 comp rx=0 tx=0).stop 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 shutdown_connections 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f6406c120 0x7f5f64133820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f6413efb0 0x7f5f64133d60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 --2- 172.21.15.133:0/1035501781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f640618a0 0x7f5f641367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 >> 172.21.15.133:0/1035501781 conn(0x7f5f640b28c0 msgr2=0x7f5f64138da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 shutdown_connections 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 -- 172.21.15.133:0/1035501781 wait complete. 2023-12-11T20:16:04.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.012+0000 7f5f69d61640 1 librados: shutdown 2023-12-11T20:16:04.013 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:04.012+0000 7fe472032040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-11T20:16:04.021 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 5830a12f-1f5b-4e09-87a1-0bb9ae893db4 7 2023-12-11T20:16:04.066 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:04.068+0000 7fe472032040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-11T20:16:04.120 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.122+0000 7f9ebca64640 10 monclient: get_monmap_and_config 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.122+0000 7f9ebca64640 10 monclient: build_initial_monmap 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.122+0000 7f9ebca64640 10 monclient: monmap: 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:16:04.123044+0000 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:16:04.123044+0000 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:04.121 INFO:teuthology.orchestra.run.smithi133.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:04.122 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:04.122 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.122+0000 7f9ebca64640 1 Processor -- start 2023-12-11T20:16:04.122 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.122+0000 7f9ebca64640 1 -- start start 2023-12-11T20:16:04.122 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.122+0000 7f9ebca64640 10 monclient: init 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9eb80a8930 0x7f9eb8145920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient(hunting): picked mon.noname-a con 0x7f9eb80a8930 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9eb80b2210 0x7f9eb813f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient(hunting): picked mon.noname-c con 0x7f9eb80b2210 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb813f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient(hunting): picked mon.noname-b con 0x7f9eb80a9b90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb813fa90 con 0x7f9eb80a8930 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb80aac10 con 0x7f9eb80b2210 2023-12-11T20:16:04.125 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8145490 con 0x7f9eb80a9b90 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.123+0000 7f9ebca64640 10 monclient(hunting): authenticate will time out at 1457.471846s 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb5d9b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9eb80b2210 0x7f9eb813f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb659c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9eb80a8930 0x7f9eb8145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb813f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb5d9b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9eb80b2210 0x7f9eb813f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:40350/0 (socket says 172.21.15.133:40350) 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb5d9b640 1 -- 172.21.15.133:0/4224214781 learned_addr learned my addr 172.21.15.133:0/4224214781 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb5d9b640 10 monclient(hunting): get_auth_request con 0x7f9eb80b2210 auth_method 0 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb5d9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb659c640 10 monclient(hunting): get_auth_request con 0x7f9eb80a8930 auth_method 0 2023-12-11T20:16:04.126 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb659c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb6d9d640 10 monclient(hunting): get_auth_request con 0x7f9eb80a9b90 auth_method 0 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb6d9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.124+0000 7f9eb6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:04.127 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.125+0000 7f9eb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 10 monclient(hunting): handle_auth_done global_id 4165 payload 274 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 1 -- 172.21.15.133:0/4224214781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 msgr2=0x7f9eb813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 1 --2- 172.21.15.133:0/4224214781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb813f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 1 -- 172.21.15.133:0/4224214781 >> v2:172.21.15.86:3300/0 conn(0x7f9eb80a8930 msgr2=0x7f9eb8145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 1 --2- 172.21.15.133:0/4224214781 >> v2:172.21.15.86:3300/0 conn(0x7f9eb80a8930 0x7f9eb8145920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 10 monclient: _finish_hunting 0 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 1 monclient: found mon.noname-c 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:04.128 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.126+0000 7f9eb5d9b640 1 -- 172.21.15.133:0/4224214781 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb80aae60 con 0x7f9eb80b2210 2023-12-11T20:16:04.129 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.128+0000 7f9eb5d9b640 1 --2- 172.21.15.133:0/4224214781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 0x7f9eb813f010 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80a8650 tx=0x7f9eac02b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cacc591230a955b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:04.131 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 1 -- 172.21.15.133:0/4224214781 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac031750 con 0x7f9eb80b2210 2023-12-11T20:16:04.131 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:04.131 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:04.131 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: dump: 2023-12-11T20:16:04.131 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:04.131 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: _finish_auth 0 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: _check_auth_tickets 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 1 -- 172.21.15.133:0/4224214781 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eac0318b0 con 0x7f9eb80b2210 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9ebca64640 5 monclient: authenticate success, global_id 4165 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9ebca64640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:04.132 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9ebca64640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 1 -- 172.21.15.133:0/4224214781 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac031c40 con 0x7f9eb80b2210 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9ebca64640 4 set_mon_vals no callback set 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9eb559a640 10 monclient: dump: 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:04.133 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9ebca64640 10 monclient: shutdown 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9ebca64640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9ebca64640 1 -- 172.21.15.133:0/4224214781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 msgr2=0x7f9eb813f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.132+0000 7f9ebca64640 1 --2- 172.21.15.133:0/4224214781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 0x7f9eb813f010 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80a8650 tx=0x7f9eac02b7f0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 1 -- 172.21.15.133:0/4224214781 shutdown_connections 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 1 --2- 172.21.15.133:0/4224214781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb813f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 1 --2- 172.21.15.133:0/4224214781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 0x7f9eb813f010 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 1 --2- 172.21.15.133:0/4224214781 >> v2:172.21.15.86:3300/0 conn(0x7f9eb80a8930 0x7f9eb8145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 1 -- 172.21.15.133:0/4224214781 >> 172.21.15.133:0/4224214781 conn(0x7f9eb80b8380 msgr2=0x7f9eb80b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 1 -- 172.21.15.133:0/4224214781 shutdown_connections 2023-12-11T20:16:04.134 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 1 -- 172.21.15.133:0/4224214781 wait complete. 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 10 monclient: build_initial_monmap 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.133+0000 7f9ebca64640 10 monclient: monmap: 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 librados: starting msgr at 2023-12-11T20:16:04.138 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 librados: starting objecter 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 Processor -- start 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 -- start start 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 librados: setting wanted keys 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 librados: calling monclient init 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient: init 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb80a8930 0x7f9eb8144390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient(hunting): picked mon.noname-a con 0x7f9eb80a8930 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient(hunting): picked mon.noname-c con 0x7f9eb80a9b90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 0x7f9eb81d6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient(hunting): picked mon.noname-b con 0x7f9eb80b2210 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:04.139 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb80aced0 con 0x7f9eb80a8930 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb80ab790 con 0x7f9eb80b2210 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb80ab340 con 0x7f9eb80a9b90 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.134+0000 7f9ebca64640 10 monclient(hunting): authenticate will time out at 1457.483229s 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb6d9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 0x7f9eb81d6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb80a8930 0x7f9eb8144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb5d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb5d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb80b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.133:36996/0 (socket says 172.21.15.133:36996) 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb5d9b640 1 -- 172.21.15.133:0/1183827985 learned_addr learned my addr 172.21.15.133:0/1183827985 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb80a8930 0x7f9eb8144390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.133:35932/0 (socket says 172.21.15.133:35932) 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb5d9b640 10 monclient(hunting): get_auth_request con 0x7f9eb80a9b90 auth_method 0 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:04.140 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb5d9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb659c640 10 monclient(hunting): get_auth_request con 0x7f9eb80a8930 auth_method 0 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb659c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb6d9d640 10 monclient(hunting): get_auth_request con 0x7f9eb80b2210 auth_method 0 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb6d9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.135+0000 7f9eb6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:04.141 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.136+0000 7f9eb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 10 monclient(hunting): handle_auth_done global_id 4136 payload 824 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 1 -- 172.21.15.133:0/1183827985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 msgr2=0x7f9eb81d6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 1 --2- 172.21.15.133:0/1183827985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 0x7f9eb81d6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 1 -- 172.21.15.133:0/1183827985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb80a8930 msgr2=0x7f9eb8144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 1 --2- 172.21.15.133:0/1183827985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb80a8930 0x7f9eb8144390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 10 monclient: _finish_hunting 0 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 1 monclient: found mon.noname-c 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb5d9b640 1 -- 172.21.15.133:0/1183827985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac0094d0 con 0x7f9eb80a9b90 2023-12-11T20:16:04.142 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.139+0000 7f9eb659c640 1 --2- 172.21.15.133:0/1183827985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb80a8930 0x7f9eb8144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.141+0000 7f9eb5d9b640 1 --2- 172.21.15.133:0/1183827985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb80b7120 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9eac02b830 tx=0x7f9eac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.141+0000 7f9ea6ffd640 10 client.?.objecter ms_handle_connect 0x7f9eb80a9b90 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.141+0000 7f9ea6ffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 1 -- 172.21.15.133:0/1183827985 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac0338d0 con 0x7f9eb80a9b90 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: dump: 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:04.144 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: _finish_auth 0 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 1 -- 172.21.15.133:0/1183827985 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eac033a50 con 0x7f9eb80a9b90 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 1 -- 172.21.15.133:0/1183827985 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac033e00 con 0x7f9eb80a9b90 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 5 monclient: authenticate success, global_id 4136 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:04.145 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ea6ffd640 10 monclient: dump: 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 10 monclient: _renew_subs 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9eb4d99640 4 set_mon_vals no callback set 2023-12-11T20:16:04.146 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb80ab500 con 0x7f9eb80a9b90 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 10 monclient: _renew_subs 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb80abe90 con 0x7f9eb80a9b90 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.144+0000 7f9ebca64640 1 librados: init done 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.145+0000 7f9ebca64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.145+0000 7f9ebca64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.145+0000 7f9ebca64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:04.147 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.145+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb804e840 con 0x7f9eb80a9b90 2023-12-11T20:16:04.148 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.148+0000 7f9ea6ffd640 1 -- 172.21.15.133:0/1183827985 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9eac03a040 con 0x7f9eb80a9b90 2023-12-11T20:16:04.148 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.148+0000 7f9ea6ffd640 1 -- 172.21.15.133:0/1183827985 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f9eac03ca60 con 0x7f9eb80a9b90 2023-12-11T20:16:04.148 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.148+0000 7f9ea6ffd640 10 client.4136.objecter ms_dispatch 0x7f9eb8000b90 osd_map(9..9 src has 1..9) v4 2023-12-11T20:16:04.148 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.148+0000 7f9ea6ffd640 3 client.4136.objecter handle_osd_map got epochs [9,9] > 0 2023-12-11T20:16:04.148 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.148+0000 7f9ea6ffd640 3 client.4136.objecter handle_osd_map decoding full epoch 9 2023-12-11T20:16:04.148 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.148+0000 7f9ea6ffd640 20 client.4136.objecter dump_active .. 0 homeless 2023-12-11T20:16:04.173 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:04.175+0000 7fe472032040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-11T20:16:04.189 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:04.190+0000 7f07fb213040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-11T20:16:04.246 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.247+0000 7f9ea6ffd640 1 -- 172.21.15.133:0/1183827985 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9eac04d060 con 0x7f9eb80a9b90 2023-12-11T20:16:04.246 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.247+0000 7f9ea6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:04.246 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.247+0000 7f9ea6ffd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:04.319 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.320+0000 7f9ebca64640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5830a12f-1f5b-4e09-87a1-0bb9ae893db4", "id": 7}] 2023-12-11T20:16:04.320 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.320+0000 7f9ebca64640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5830a12f-1f5b-4e09-87a1-0bb9ae893db4", "id": 7}] 2023-12-11T20:16:04.320 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.320+0000 7f9ebca64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:04.320 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.320+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5830a12f-1f5b-4e09-87a1-0bb9ae893db4", "id": 7} v 0) v1 -- 0x7f9eb80b2f40 con 0x7f9eb80a9b90 2023-12-11T20:16:04.479 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:04.480+0000 7f07fb213040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-11T20:16:04.532 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:04.533+0000 7f07fb213040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-11T20:16:04.539 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:04.540+0000 7f9ea6ffd640 1 -- 172.21.15.133:0/1183827985 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mgrmap(e 2) v1 ==== 44567+0+0 (secure 0 0 0) 0x7f9eac03cd40 con 0x7f9eb80a9b90 2023-12-11T20:16:04.636 INFO:tasks.ceph.mgr.y.smithi086.stderr:2023-12-11T20:16:04.637+0000 7f07fb213040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-11T20:16:05.021 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.022+0000 7f9ea6ffd640 1 -- 172.21.15.133:0/1183827985 <== mon.2 v2:172.21.15.86:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5830a12f-1f5b-4e09-87a1-0bb9ae893db4", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9eac050d20 con 0x7f9eb80a9b90 2023-12-11T20:16:05.021 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.022+0000 7f9ea6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5830a12f-1f5b-4e09-87a1-0bb9ae893db4", "id": 7}] 2023-12-11T20:16:05.021 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.022+0000 7f9ea6ffd640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:05.021 INFO:teuthology.orchestra.run.smithi133.stdout:7 2023-12-11T20:16:05.022 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 10 librados: watch_flush enter 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 10 librados: watch_flush exit 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 20 client.4136.objecter shutdown clearing up homeless session... 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 10 client.4136.objecter successfully canceled tick 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 10 monclient: shutdown 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 msgr2=0x7f9eb80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 --2- 172.21.15.133:0/1183827985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb80b7120 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9eac02b830 tx=0x7f9eac0046c0 comp rx=0 tx=0).stop 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 shutdown_connections 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 --2- 172.21.15.133:0/1183827985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eb80a9b90 0x7f9eb80b7120 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 --2- 172.21.15.133:0/1183827985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9eb80b2210 0x7f9eb81d6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 --2- 172.21.15.133:0/1183827985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb80a8930 0x7f9eb8144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 >> 172.21.15.133:0/1183827985 conn(0x7f9eb80b8380 msgr2=0x7f9eb813b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 shutdown_connections 2023-12-11T20:16:05.023 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 -- 172.21.15.133:0/1183827985 wait complete. 2023-12-11T20:16:05.024 INFO:teuthology.orchestra.run.smithi133.stderr:2023-12-11T20:16:05.023+0000 7f9ebca64640 1 librados: shutdown 2023-12-11T20:16:05.032 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-11T20:16:05.032 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-11T20:16:05.032 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-12-11T20:16:05.034 INFO:tasks.ceph.osd.0:Started 2023-12-11T20:16:05.034 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-11T20:16:05.034 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-11T20:16:05.035 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-12-11T20:16:05.036 INFO:tasks.ceph.osd.1:Started 2023-12-11T20:16:05.037 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-11T20:16:05.037 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-11T20:16:05.037 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-12-11T20:16:05.041 INFO:tasks.ceph.osd.2:Started 2023-12-11T20:16:05.041 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-11T20:16:05.041 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-11T20:16:05.041 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-12-11T20:16:05.043 INFO:tasks.ceph.osd.3:Started 2023-12-11T20:16:05.044 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-11T20:16:05.044 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-11T20:16:05.044 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-12-11T20:16:05.046 INFO:tasks.ceph.osd.4:Started 2023-12-11T20:16:05.046 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-11T20:16:05.046 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-11T20:16:05.046 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-12-11T20:16:05.048 INFO:tasks.ceph.osd.5:Started 2023-12-11T20:16:05.048 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-11T20:16:05.049 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-11T20:16:05.049 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-12-11T20:16:05.051 INFO:tasks.ceph.osd.6:Started 2023-12-11T20:16:05.051 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-11T20:16:05.051 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-11T20:16:05.051 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-12-11T20:16:05.053 INFO:tasks.ceph.osd.7:Started 2023-12-11T20:16:05.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.214+0000 7fa76ac90640 10 monclient: get_monmap_and_config 2023-12-11T20:16:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.214+0000 7fa76ac90640 10 monclient: build_initial_monmap 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.214+0000 7fa76ac90640 10 monclient: monmap: 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:05.215945+0000 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:05.215945+0000 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:05.214 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.215+0000 7fa76ac90640 1 Processor -- start 2023-12-11T20:16:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.215+0000 7fa76ac90640 1 -- start start 2023-12-11T20:16:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.215+0000 7fa76ac90640 10 monclient: init 2023-12-11T20:16:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-12-11T20:16:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7640a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fa7641461c0 0x7fa7640a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 10 monclient(hunting): picked mon.noname-c con 0x7fa7641461c0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa76413f0f0 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa764145620 con 0x7fa76413f0f0 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7640aace0 con 0x7fa7640a8a20 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76413fc60 con 0x7fa7641461c0 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.216+0000 7fa76ac90640 10 monclient(hunting): authenticate will time out at 1451.546838s 2023-12-11T20:16:05.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.237+0000 7fa76922d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa76413f0f0 0x7fa76413f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46500/0 (socket says 172.21.15.86:46500) 2023-12-11T20:16:05.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.237+0000 7fa76922d640 1 -- 172.21.15.86:0/343565609 learned_addr learned my addr 172.21.15.86:0/343565609 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:16:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.246+0000 7fa763fff640 1 --2- 172.21.15.86:0/343565609 >> v2:172.21.15.133:3300/0 conn(0x7fa7641461c0 0x7fa7640a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.246+0000 7fa763fff640 10 monclient(hunting): get_auth_request con 0x7fa7641461c0 auth_method 0 2023-12-11T20:16:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.246+0000 7fa763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.246+0000 7fa763fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.246+0000 7fa763fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.247+0000 7fa763fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.247+0000 7fa763fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.247+0000 7fa763fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa7637fe640 1 -- 172.21.15.86:0/343565609 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099275164 0 0) 0x7fa764145620 con 0x7fa76413f0f0 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa7637fe640 10 monclient(hunting): _negotiate 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa7637fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa7637fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa7637fe640 10 monclient(hunting): my global_id is 4158 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa7637fe640 1 -- 172.21.15.86:0/343565609 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa754003520 con 0x7fa76413f0f0 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa763fff640 10 monclient(hunting): handle_auth_done global_id 4189 payload 274 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa763fff640 1 -- 172.21.15.86:0/343565609 >> v1:172.21.15.133:6789/0 conn(0x7fa7640a8a20 legacy=0x7fa764145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa763fff640 1 -- 172.21.15.86:0/343565609 >> v1:172.21.15.86:6789/0 conn(0x7fa76413f0f0 legacy=0x7fa76413f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa763fff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa763fff640 1 monclient: found mon.noname-c 2023-12-11T20:16:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa763fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:05.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa763fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:05.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.249+0000 7fa763fff640 1 -- 172.21.15.86:0/343565609 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7640aafc0 con 0x7fa7641461c0 2023-12-11T20:16:05.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.256+0000 7fa763fff640 1 --2- 172.21.15.86:0/343565609 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 0x7fa7640a8e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa7500097a0 tx=0x7fa75002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b571826e889106bf server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:05.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 1 -- 172.21.15.86:0/343565609 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750030610 con 0x7fa7641461c0 2023-12-11T20:16:05.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:05.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:05.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: dump: 2023-12-11T20:16:05.266 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:05.266 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:05.266 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: _finish_auth 0 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: _check_auth_tickets 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 1 -- 172.21.15.86:0/343565609 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa750030770 con 0x7fa7641461c0 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 1 -- 172.21.15.86:0/343565609 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750030b00 con 0x7fa7641461c0 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa76ac90640 5 monclient: authenticate success, global_id 4189 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa7637fe640 10 monclient: dump: 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:05.268 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa76ac90640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.265+0000 7fa76ac90640 4 set_mon_vals no callback set 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.266+0000 7fa76ac90640 10 monclient: shutdown 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.266+0000 7fa76ac90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.266+0000 7fa76ac90640 1 -- 172.21.15.86:0/343565609 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 msgr2=0x7fa7640a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.266+0000 7fa76ac90640 1 --2- 172.21.15.86:0/343565609 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 0x7fa7640a8e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa7500097a0 tx=0x7fa75002b960 comp rx=0 tx=0).stop 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.266+0000 7fa76ac90640 1 -- 172.21.15.86:0/343565609 shutdown_connections 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.266+0000 7fa76ac90640 1 --2- 172.21.15.86:0/343565609 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 0x7fa7640a8e00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.266+0000 7fa76ac90640 1 -- 172.21.15.86:0/343565609 >> 172.21.15.86:0/343565609 conn(0x7fa7640b7890 msgr2=0x7fa7640b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.267+0000 7fa76ac90640 1 -- 172.21.15.86:0/343565609 shutdown_connections 2023-12-11T20:16:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.267+0000 7fa76ac90640 1 -- 172.21.15.86:0/343565609 wait complete. 2023-12-11T20:16:05.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.267+0000 7fa76ac90640 10 monclient: build_initial_monmap 2023-12-11T20:16:05.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.267+0000 7fa76ac90640 10 monclient: monmap: 2023-12-11T20:16:05.273 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:05.273 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:05.273 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:05.273 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:05.273 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.268+0000 7fa76ac90640 1 librados: starting msgr at 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.268+0000 7fa76ac90640 1 librados: starting objecter 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.268+0000 7fa76ac90640 1 Processor -- start 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.268+0000 7fa76ac90640 1 -- start start 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.268+0000 7fa76ac90640 1 librados: setting wanted keys 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.268+0000 7fa76ac90640 1 librados: calling monclient init 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.268+0000 7fa76ac90640 10 monclient: init 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.269+0000 7fa76ac90640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:05.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.269+0000 7fa76ac90640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.269+0000 7fa76ac90640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7640a8a20 0x7fa7641401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.269+0000 7fa76ac90640 10 monclient(hunting): picked mon.noname-c con 0x7fa7640a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.269+0000 7fa76ac90640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76413f0f0 0x7fa764142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 10 monclient(hunting): picked mon.noname-a con 0x7fa76413f0f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 0x7fa764140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 10 monclient(hunting): picked mon.noname-b con 0x7fa7641461c0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa764140a50 con 0x7fa76413f0f0 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa764141420 con 0x7fa7641461c0 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa768a2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7640a8a20 0x7fa7641401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa764142070 con 0x7fa7640a8a20 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76ac90640 10 monclient(hunting): authenticate will time out at 1451.600851s 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa76922d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 0x7fa764140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa768a2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7640a8a20 0x7fa7641401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58668/0 (socket says 172.21.15.86:58668) 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa768a2c640 1 -- 172.21.15.86:0/3537740599 learned_addr learned my addr 172.21.15.86:0/3537740599 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa768a2c640 10 monclient(hunting): get_auth_request con 0x7fa7640a8a20 auth_method 0 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa768a2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa768a2c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.270+0000 7fa768a2c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa76922d640 10 monclient(hunting): get_auth_request con 0x7fa7641461c0 auth_method 0 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa76922d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa76922d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:05.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa76922d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:05.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa768a2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:05.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa768a2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:05.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa768a2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:05.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa76922d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:05.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa76922d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:05.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.271+0000 7fa76922d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:05.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 10 monclient(hunting): handle_auth_done global_id 4148 payload 824 2023-12-11T20:16:05.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 1 -- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 msgr2=0x7fa764140ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:05.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 0x7fa764140ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 1 -- 172.21.15.86:0/3537740599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76413f0f0 msgr2=0x7fa764142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76413f0f0 0x7fa764142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 10 monclient: _finish_hunting 0 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 1 monclient: found mon.noname-c 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.275+0000 7fa768a2c640 1 -- 172.21.15.86:0/3537740599 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa750009400 con 0x7fa7640a8a20 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.279+0000 7fa768a2c640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7640a8a20 0x7fa7641401d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa75800c3c0 tx=0x7fa75800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.279+0000 7fa7617fa640 10 client.?.objecter ms_handle_connect 0x7fa7640a8a20 2023-12-11T20:16:05.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.279+0000 7fa7617fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:05.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 1 -- 172.21.15.86:0/3537740599 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7580052d0 con 0x7fa7640a8a20 2023-12-11T20:16:05.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:05.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 10 monclient: dump: 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 10 monclient: _finish_auth 0 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 10 monclient: _check_auth_tickets 2023-12-11T20:16:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 1 -- 172.21.15.86:0/3537740599 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa758005450 con 0x7fa7640a8a20 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa7617fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.284+0000 7fa762ffd640 4 set_mon_vals no callback set 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.285+0000 7fa76ac90640 5 monclient: authenticate success, global_id 4148 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.285+0000 7fa7617fa640 1 -- 172.21.15.86:0/3537740599 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758007260 con 0x7fa7640a8a20 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.285+0000 7fa76ac90640 10 monclient: _renew_subs 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.285+0000 7fa76ac90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.285+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa764142350 con 0x7fa7640a8a20 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.286+0000 7fa7617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.286+0000 7fa7617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.286+0000 7fa7617fa640 10 monclient: dump: 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.286+0000 7fa76ac90640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.286+0000 7fa76ac90640 10 monclient: _renew_subs 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.286+0000 7fa76ac90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.286+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa764143000 con 0x7fa7640a8a20 2023-12-11T20:16:05.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.286+0000 7fa76ac90640 1 librados: init done 2023-12-11T20:16:05.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.287+0000 7fa76ac90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:05.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.287+0000 7fa76ac90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:05.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.287+0000 7fa76ac90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:05.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.287+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76404e840 con 0x7fa7640a8a20 2023-12-11T20:16:05.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.341+0000 7fa7617fa640 1 -- 172.21.15.86:0/3537740599 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 2) v1 ==== 44567+0+0 (secure 0 0 0) 0x7fa7580073c0 con 0x7fa7640a8a20 2023-12-11T20:16:05.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.466+0000 7fa7617fa640 1 -- 172.21.15.86:0/3537740599 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa758019dc0 con 0x7fa7640a8a20 2023-12-11T20:16:05.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.466+0000 7fa7617fa640 10 client.4148.objecter ms_dispatch 0x7fa764000b90 osd_map(10..10 src has 1..10) v4 2023-12-11T20:16:05.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.466+0000 7fa7617fa640 3 client.4148.objecter handle_osd_map got epochs [10,10] > 0 2023-12-11T20:16:05.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.466+0000 7fa7617fa640 3 client.4148.objecter handle_osd_map decoding full epoch 10 2023-12-11T20:16:05.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.466+0000 7fa7617fa640 20 client.4148.objecter dump_active .. 0 homeless 2023-12-11T20:16:05.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.466+0000 7fa7617fa640 1 -- 172.21.15.86:0/3537740599 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa758007670 con 0x7fa7640a8a20 2023-12-11T20:16:05.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.466+0000 7fa7617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:05.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.466+0000 7fa7617fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.589+0000 7fa76ac90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.589+0000 7fa76ac90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.589+0000 7fa76ac90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.589+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa76413dc90 con 0x7fa7640a8a20 2023-12-11T20:16:05.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.884+0000 7fa7617fa640 1 -- 172.21.15.86:0/3537740599 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mgrmap(e 3) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fa758012d20 con 0x7fa7640a8a20 2023-12-11T20:16:05.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.884+0000 7fa7617fa640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa74806cd60 0x7fa74806f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:05.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.885+0000 7fa763fff640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa74806cd60 0x7fa74806f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:05.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.885+0000 7fa763fff640 10 monclient: get_auth_request con 0x7fa74806cd60 auth_method 0 2023-12-11T20:16:05.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.885+0000 7fa7617fa640 10 client.4148.objecter ms_handle_connect 0x7fa74806cd60 2023-12-11T20:16:05.884 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:05.885+0000 7fa763fff640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa74806cd60 0x7fa74806f150 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa750036000 tx=0x7fa75003b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:06.031 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:06.033+0000 7fe412c42640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-11T20:16:06.111 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.111+0000 7fa7617fa640 1 -- 172.21.15.86:0/3537740599 <== mon.2 v2:172.21.15.86:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa758056390 con 0x7fa7640a8a20 2023-12-11T20:16:06.111 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.111+0000 7fa7617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:06.111 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.111+0000 7fa7617fa640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:16:06.111 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":10,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:04.659077+0000","last_up_change":"0.000000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:06.112 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 10 librados: watch_flush enter 2023-12-11T20:16:06.112 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 10 librados: watch_flush exit 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 20 client.4148.objecter shutdown clearing up homeless session... 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 10 client.4148.objecter successfully canceled tick 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa74806cd60 msgr2=0x7fa74806f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa74806cd60 0x7fa74806f150 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa750036000 tx=0x7fa75003b040 comp rx=0 tx=0).stop 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 10 monclient: shutdown 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7640a8a20 msgr2=0x7fa7641401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7640a8a20 0x7fa7641401d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa75800c3c0 tx=0x7fa75800c880 comp rx=0 tx=0).stop 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 shutdown_connections 2023-12-11T20:16:06.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa74806cd60 0x7fa74806f150 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:06.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7641461c0 0x7fa764140ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:06.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76413f0f0 0x7fa764142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:06.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 --2- 172.21.15.86:0/3537740599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7640a8a20 0x7fa7641401d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:06.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 >> 172.21.15.86:0/3537740599 conn(0x7fa7640b7890 msgr2=0x7fa7640b7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:06.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 shutdown_connections 2023-12-11T20:16:06.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 -- 172.21.15.86:0/3537740599 wait complete. 2023-12-11T20:16:06.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:06.113+0000 7fa76ac90640 1 librados: shutdown 2023-12-11T20:16:06.128 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-11T20:16:06.128 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-11T20:16:06.197 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:16:06.198+0000 7fe412c42640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-11T20:16:06.935 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:07.078 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.079+0000 7f674a0af640 10 monclient: get_monmap_and_config 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.079+0000 7f674a0af640 10 monclient: build_initial_monmap 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.079+0000 7f674a0af640 10 monclient: monmap: 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:16:07.081163+0000 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:16:07.081163+0000 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:07.079 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:07.080 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:07.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.080+0000 7f674a0af640 1 Processor -- start 2023-12-11T20:16:07.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.080+0000 7f674a0af640 1 -- start start 2023-12-11T20:16:07.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.080+0000 7f674a0af640 10 monclient: init 2023-12-11T20:16:07.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:07.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-12-11T20:16:07.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f67440a8900 0x7f674413efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:07.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient(hunting): picked mon.noname-c con 0x7f67440a8900 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:07.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67440a9b80 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:07.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f67440a8ce0 0x7f67441453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient(hunting): picked mon.noname-a con 0x7f67440a8ce0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67440b1b80 con 0x7f67440a9b80 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67440aa870 con 0x7f67440a8ce0 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67440aab50 con 0x7f67440a8900 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.081+0000 7f674a0af640 10 monclient(hunting): authenticate will time out at 1453.412062s 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.082+0000 7f6743fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f67440a8ce0 0x7f67441453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.082+0000 7f67437fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f67440a8900 0x7f674413efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f67440a8ce0 0x7f67441453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46474/0 (socket says 172.21.15.86:46474) 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 1 -- 172.21.15.86:0/267343410 learned_addr learned my addr 172.21.15.86:0/267343410 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:07.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 10 monclient(hunting): get_auth_request con 0x7f67440a8ce0 auth_method 0 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f67437fe640 10 monclient(hunting): get_auth_request con 0x7f67440a8900 auth_method 0 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f67437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f67437fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f67437fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.083+0000 7f6743fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.084+0000 7f67437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.084+0000 7f67437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:07.086 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.084+0000 7f67437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 10 monclient(hunting): handle_auth_done global_id 4161 payload 274 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 1 -- 172.21.15.86:0/267343410 >> v2:172.21.15.133:3300/0 conn(0x7f67440a8900 msgr2=0x7f674413efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 1 --2- 172.21.15.86:0/267343410 >> v2:172.21.15.133:3300/0 conn(0x7f67440a8900 0x7f674413efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 1 -- 172.21.15.86:0/267343410 >> v1:172.21.15.133:6789/0 conn(0x7f67440a9b80 legacy=0x7f674413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 1 monclient: found mon.noname-a 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.086+0000 7f6743fff640 1 -- 172.21.15.86:0/267343410 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6744145930 con 0x7f67440a8ce0 2023-12-11T20:16:07.088 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.089+0000 7f6743fff640 1 --2- 172.21.15.86:0/267343410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 0x7f67441453f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67440b4fb0 tx=0x7f673c02bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d91672413cfee72 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:07.095 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 1 -- 172.21.15.86:0/267343410 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c02cb30 con 0x7f67440a8ce0 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: dump: 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:07.096 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 1 -- 172.21.15.86:0/267343410 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673c02cc90 con 0x7f67440a8ce0 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 1 -- 172.21.15.86:0/267343410 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c034610 con 0x7f67440a8ce0 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f67427fc640 10 monclient: dump: 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:07.097 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f674a0af640 5 monclient: authenticate success, global_id 4161 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f674a0af640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.094+0000 7f674a0af640 4 set_mon_vals no callback set 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.095+0000 7f674a0af640 10 monclient: shutdown 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.095+0000 7f674a0af640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.095+0000 7f674a0af640 1 -- 172.21.15.86:0/267343410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 msgr2=0x7f67441453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.095+0000 7f674a0af640 1 --2- 172.21.15.86:0/267343410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 0x7f67441453f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67440b4fb0 tx=0x7f673c02bf90 comp rx=0 tx=0).stop 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.096+0000 7f674a0af640 1 -- 172.21.15.86:0/267343410 shutdown_connections 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.096+0000 7f674a0af640 1 --2- 172.21.15.86:0/267343410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 0x7f67441453f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:07.098 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.096+0000 7f674a0af640 1 --2- 172.21.15.86:0/267343410 >> v2:172.21.15.133:3300/0 conn(0x7f67440a8900 0x7f674413efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:07.099 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.096+0000 7f674a0af640 1 -- 172.21.15.86:0/267343410 >> 172.21.15.86:0/267343410 conn(0x7f6744066f00 msgr2=0x7f67440b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:07.099 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.096+0000 7f674a0af640 1 -- 172.21.15.86:0/267343410 shutdown_connections 2023-12-11T20:16:07.099 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.096+0000 7f674a0af640 1 -- 172.21.15.86:0/267343410 wait complete. 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.097+0000 7f674a0af640 10 monclient: build_initial_monmap 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.097+0000 7f674a0af640 10 monclient: monmap: 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:07.104 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.098+0000 7f674a0af640 1 librados: starting msgr at 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.098+0000 7f674a0af640 1 librados: starting objecter 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.098+0000 7f674a0af640 1 Processor -- start 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.098+0000 7f674a0af640 1 -- start start 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.098+0000 7f674a0af640 1 librados: setting wanted keys 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.098+0000 7f674a0af640 1 librados: calling monclient init 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.098+0000 7f674a0af640 10 monclient: init 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67440a8900 0x7f67440b6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 10 monclient(hunting): picked mon.noname-b con 0x7f67440a8900 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 0x7f67440b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 10 monclient(hunting): picked mon.noname-a con 0x7f67440a8ce0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67440a9b80 0x7f67441debb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 10 monclient(hunting): picked mon.noname-c con 0x7f67440a9b80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:07.105 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67440b83a0 con 0x7f67440a8ce0 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.099+0000 7f674a0af640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67440b7870 con 0x7f67440a8900 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f674a0af640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f674a0af640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67440b73e0 con 0x7f67440a9b80 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f674a0af640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f674a0af640 10 monclient(hunting): authenticate will time out at 1453.430134s 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f67437fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67440a8900 0x7f67440b6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f6742ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 0x7f67440b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f6743fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67440a9b80 0x7f67441debb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f6742ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 0x7f67440b6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46476/0 (socket says 172.21.15.86:46476) 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f6742ffd640 1 -- 172.21.15.86:0/3541889822 learned_addr learned my addr 172.21.15.86:0/3541889822 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f6742ffd640 10 monclient(hunting): get_auth_request con 0x7f67440a8ce0 auth_method 0 2023-12-11T20:16:07.106 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f6742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f6742ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f6742ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f67437fe640 10 monclient(hunting): get_auth_request con 0x7f67440a8900 auth_method 0 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f67437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f67437fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.100+0000 7f67437fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6743fff640 10 monclient(hunting): get_auth_request con 0x7f67440a9b80 auth_method 0 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6743fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6743fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6743fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6742ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6742ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6742ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f67437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f67437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:07.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f67437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:07.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6743fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:07.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6743fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:07.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.101+0000 7f6743fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:07.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 10 monclient(hunting): handle_auth_done global_id 4204 payload 824 2023-12-11T20:16:07.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 1 -- 172.21.15.86:0/3541889822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67440a9b80 msgr2=0x7f67441debb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:07.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67440a9b80 0x7f67441debb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 1 -- 172.21.15.86:0/3541889822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 msgr2=0x7f67440b6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 0x7f67440b6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 10 monclient: _finish_hunting 0 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 1 monclient: found mon.noname-b 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.105+0000 7f67437fe640 1 -- 172.21.15.86:0/3541889822 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c009b30 con 0x7f67440a8900 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.109+0000 7f67437fe640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67440a8900 0x7f67440b6d50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f673400e8d0 tx=0x7f673400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.109+0000 7f672bfff640 10 client.?.objecter ms_handle_connect 0x7f67440a8900 2023-12-11T20:16:07.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.109+0000 7f672bfff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:07.112 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f672bfff640 1 -- 172.21.15.86:0/3541889822 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734009e60 con 0x7f67440a8900 2023-12-11T20:16:07.112 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f672bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:07.112 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f672bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:07.112 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f672bfff640 10 monclient: dump: 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f672bfff640 10 monclient: _finish_auth 0 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f672bfff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f672bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:07.113 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 5 monclient: authenticate success, global_id 4204 2023-12-11T20:16:07.114 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 10 monclient: _renew_subs 2023-12-11T20:16:07.114 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:07.114 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674406dad0 con 0x7f67440a8900 2023-12-11T20:16:07.114 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:07.114 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 10 monclient: _renew_subs 2023-12-11T20:16:07.114 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:07.114 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67441df2d0 con 0x7f67440a8900 2023-12-11T20:16:07.114 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.112+0000 7f674a0af640 1 librados: init done 2023-12-11T20:16:07.115 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f674a0af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:07.115 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f674a0af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f674a0af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674406b320 con 0x7f67440a8900 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f672bfff640 1 -- 172.21.15.86:0/3541889822 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6734005990 con 0x7f67440a8900 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f672bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f672bfff640 1 -- 172.21.15.86:0/3541889822 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734005d00 con 0x7f67440a8900 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f672bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f672bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f672bfff640 10 monclient: dump: 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:07.116 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:07.117 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:07.117 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:07.117 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:07.117 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:07.117 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.114+0000 7f6741ffb640 4 set_mon_vals no callback set 2023-12-11T20:16:07.559 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.560+0000 7f672bfff640 1 -- 172.21.15.86:0/3541889822 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 3) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f673400c940 con 0x7f67440a8900 2023-12-11T20:16:07.560 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.561+0000 7f672bfff640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f673006c670 0x7f673006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:07.560 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.561+0000 7f6742ffd640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f673006c670 0x7f673006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:07.560 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.561+0000 7f6742ffd640 10 monclient: get_auth_request con 0x7f673006c670 auth_method 0 2023-12-11T20:16:07.561 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.562+0000 7f6742ffd640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f673006c670 0x7f673006eb30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f67440b1720 tx=0x7f6738009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:07.561 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.562+0000 7f672bfff640 10 client.4204.objecter ms_handle_connect 0x7f673006c670 2023-12-11T20:16:07.564 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.564+0000 7f672bfff640 1 -- 172.21.15.86:0/3541889822 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6734058b00 con 0x7f67440a8900 2023-12-11T20:16:07.564 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.564+0000 7f672bfff640 10 client.4204.objecter ms_dispatch 0x7f6744000b90 osd_map(10..10 src has 1..10) v4 2023-12-11T20:16:07.564 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.564+0000 7f672bfff640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2023-12-11T20:16:07.564 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.564+0000 7f672bfff640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2023-12-11T20:16:07.564 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.564+0000 7f672bfff640 20 client.4204.objecter dump_active .. 0 homeless 2023-12-11T20:16:07.587 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.588+0000 7f672bfff640 1 -- 172.21.15.86:0/3541889822 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6734054e40 con 0x7f67440a8900 2023-12-11T20:16:07.587 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.588+0000 7f672bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:07.587 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.588+0000 7f672bfff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:07.892 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.892+0000 7f672bfff640 1 -- 172.21.15.86:0/3541889822 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f6734055210 con 0x7f67440a8900 2023-12-11T20:16:07.922 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.923+0000 7f674a0af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:07.922 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.923+0000 7f674a0af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:07.922 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.923+0000 7f674a0af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:07.923 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:07.923+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67440681d0 con 0x7f67440a8900 2023-12-11T20:16:08.079 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.079+0000 7f672bfff640 1 -- 172.21.15.86:0/3541889822 <== mon.1 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f67340762a0 con 0x7f67440a8900 2023-12-11T20:16:08.079 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.079+0000 7f672bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:08.079 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.079+0000 7f672bfff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:08.079 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-11T20:16:08.079 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:04.659077+0000","last_up_change":"0.000000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 10 librados: watch_flush enter 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 10 librados: watch_flush exit 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 20 client.4204.objecter shutdown clearing up homeless session... 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 10 client.4204.objecter successfully canceled tick 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f673006c670 msgr2=0x7f673006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f673006c670 0x7f673006eb30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f67440b1720 tx=0x7f6738009040 comp rx=0 tx=0).stop 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 10 monclient: shutdown 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:08.083 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67440a8900 msgr2=0x7f67440b6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:08.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67440a8900 0x7f67440b6d50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f673400e8d0 tx=0x7f673400ed90 comp rx=0 tx=0).stop 2023-12-11T20:16:08.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 shutdown_connections 2023-12-11T20:16:08.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f673006c670 0x7f673006eb30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:08.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67440a9b80 0x7f67441debb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:08.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67440a8ce0 0x7f67440b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:08.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 --2- 172.21.15.86:0/3541889822 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67440a8900 0x7f67440b6d50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:08.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 >> 172.21.15.86:0/3541889822 conn(0x7f6744066f00 msgr2=0x7f67440b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:08.084 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 shutdown_connections 2023-12-11T20:16:08.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 -- 172.21.15.86:0/3541889822 wait complete. 2023-12-11T20:16:08.085 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:08.084+0000 7f674a0af640 1 librados: shutdown 2023-12-11T20:16:08.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-11T20:16:13.041 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:07.930 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.066 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:07.949 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.107 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:07.996 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.124 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:08.014 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.126 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:08.010 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.184 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:08.074 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.293 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:08.164 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.354 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:08.225 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.445 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:08.339 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.496 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:08.393 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.497 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:08.384 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.504 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:08.398 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.513 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:08.398 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.555 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:08.451 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.555 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:08.443 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.574 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:08.459 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:13.656 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:16:13.652+0000 5dc2340 -1 Falling back to public interface 2023-12-11T20:16:13.682 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:16:13.677+0000 5dc2340 -1 Falling back to public interface 2023-12-11T20:16:13.733 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:16:13.729+0000 5dc2340 -1 Falling back to public interface 2023-12-11T20:16:13.898 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:16:13.894+0000 5dc2340 -1 Falling back to public interface 2023-12-11T20:16:14.064 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:16:14.060+0000 5dc2340 -1 Falling back to public interface 2023-12-11T20:16:14.098 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:08.987 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.108 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:16:14.104+0000 5dc2340 -1 Falling back to public interface 2023-12-11T20:16:14.134 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:16:14.125+0000 5dc2340 -1 Falling back to public interface 2023-12-11T20:16:14.142 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:09.026 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.164 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:16:14.160+0000 5dc2340 -1 Falling back to public interface 2023-12-11T20:16:14.174 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:09.065 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.366 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:09.237 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.459 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:09.348 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.505 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:09.389 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.517 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:09.410 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.571 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:09.461 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.585 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:09.471 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.616 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:09.512 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.634 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:09.520 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.725 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:09.596 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.816 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:09.705 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.868 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:09.752 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.874 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:09.768 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.909 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:14.936 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:09.826 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.954 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:09.841 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.985 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:09.881 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.987 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:09.872 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:14.997 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:14.999+0000 7f10fd84b640 10 monclient: get_monmap_and_config 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:14.999+0000 7f10fd84b640 10 monclient: build_initial_monmap 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:14.999+0000 7f10fd84b640 10 monclient: monmap: 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:16:15.000231+0000 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:16:15.000231+0000 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:14.998 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:14.999 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:14.999 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:15.002 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.000+0000 7f10fd84b640 1 Processor -- start 2023-12-11T20:16:15.002 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.000+0000 7f10fd84b640 1 -- start start 2023-12-11T20:16:15.002 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.000+0000 7f10fd84b640 10 monclient: init 2023-12-11T20:16:15.002 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:15.002 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-12-11T20:16:15.002 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f10f80a8900 0x7f10f813efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:15.003 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient(hunting): picked mon.noname-c con 0x7f10f80a8900 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:15.003 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10f80a9b80 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:15.003 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f81453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:15.003 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient(hunting): picked mon.noname-b con 0x7f10f80a8ce0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:15.003 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80b1b80 con 0x7f10f80a9b80 2023-12-11T20:16:15.003 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:15.004 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f80aa870 con 0x7f10f80a8900 2023-12-11T20:16:15.004 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:15.004 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80aab50 con 0x7f10f80a8ce0 2023-12-11T20:16:15.004 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:15.004 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.001+0000 7f10fd84b640 10 monclient(hunting): authenticate will time out at 1461.330459s 2023-12-11T20:16:15.004 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f6ffd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f10f80a8900 0x7f10f813efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:15.004 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f81453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:15.005 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f81453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53792/0 (socket says 172.21.15.86:53792) 2023-12-11T20:16:15.005 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 1 -- 172.21.15.86:0/3605075473 learned_addr learned my addr 172.21.15.86:0/3605075473 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:15.005 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 10 monclient(hunting): get_auth_request con 0x7f10f80a8ce0 auth_method 0 2023-12-11T20:16:15.005 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:15.005 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:15.005 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f6ffd640 10 monclient(hunting): get_auth_request con 0x7f10f80a8900 auth_method 0 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.002+0000 7f10f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.003+0000 7f10f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:15.006 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.003+0000 7f10f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:15.007 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.003+0000 7f10f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 1 -- 172.21.15.86:0/3605075473 >> v2:172.21.15.133:3300/0 conn(0x7f10f80a8900 msgr2=0x7f10f813efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 1 --2- 172.21.15.86:0/3605075473 >> v2:172.21.15.133:3300/0 conn(0x7f10f80a8900 0x7f10f813efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 1 -- 172.21.15.86:0/3605075473 >> v1:172.21.15.133:6789/0 conn(0x7f10f80a9b80 legacy=0x7f10f813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 10 monclient: _finish_hunting 0 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 1 monclient: found mon.noname-b 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.004+0000 7f10f77fe640 1 -- 172.21.15.86:0/3605075473 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f8145930 con 0x7f10f80a8ce0 2023-12-11T20:16:15.012 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.006+0000 7f10f77fe640 1 --2- 172.21.15.86:0/3605075473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f81453f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009e50 tx=0x7f10f002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c574d60b5d50db35 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 1 -- 172.21.15.86:0/3605075473 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f002cb00 con 0x7f10f80a8ce0 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: dump: 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:15.013 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 1 -- 172.21.15.86:0/3605075473 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f002cc60 con 0x7f10f80a8ce0 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 5 monclient: authenticate success, global_id 4172 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 4 set_mon_vals no callback set 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 1 -- 172.21.15.86:0/3605075473 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f00343f0 con 0x7f10f80a8ce0 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10f67fc640 10 monclient: dump: 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:15.014 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 10 monclient: shutdown 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 1 -- 172.21.15.86:0/3605075473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 msgr2=0x7f10f81453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 1 --2- 172.21.15.86:0/3605075473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f81453f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009e50 tx=0x7f10f002bf60 comp rx=0 tx=0).stop 2023-12-11T20:16:15.015 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 1 -- 172.21.15.86:0/3605075473 shutdown_connections 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 1 --2- 172.21.15.86:0/3605075473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f81453f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 1 --2- 172.21.15.86:0/3605075473 >> v2:172.21.15.133:3300/0 conn(0x7f10f80a8900 0x7f10f813efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 1 -- 172.21.15.86:0/3605075473 >> 172.21.15.86:0/3605075473 conn(0x7f10f8066f00 msgr2=0x7f10f80b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 1 -- 172.21.15.86:0/3605075473 shutdown_connections 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.010+0000 7f10fd84b640 1 -- 172.21.15.86:0/3605075473 wait complete. 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 10 monclient: build_initial_monmap 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 10 monclient: monmap: 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:15.016 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 1 librados: starting msgr at 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 1 librados: starting objecter 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 1 Processor -- start 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 1 -- start start 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 1 librados: setting wanted keys 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 1 librados: calling monclient init 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.011+0000 7f10fd84b640 10 monclient: init 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10f80a8900 0x7f10f81427d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient(hunting): picked mon.noname-b con 0x7f10f80a8900 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f8146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:15.017 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient(hunting): picked mon.noname-c con 0x7f10f80a8ce0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10f80a9b80 0x7f10f81d76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient(hunting): picked mon.noname-a con 0x7f10f80a9b80 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8141330 con 0x7f10f80a9b80 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f81466c0 con 0x7f10f80a8900 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8147ec0 con 0x7f10f80a8ce0 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.012+0000 7f10fd84b640 10 monclient(hunting): authenticate will time out at 1461.341620s 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f8146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10f80a8900 0x7f10f81427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10f80a9b80 0x7f10f81d76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:15.018 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f8146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53806/0 (socket says 172.21.15.86:53806) 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10f80a8900 0x7f10f81427d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:59072/0 (socket says 172.21.15.86:59072) 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 1 -- 172.21.15.86:0/2151603106 learned_addr learned my addr 172.21.15.86:0/2151603106 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 10 monclient(hunting): get_auth_request con 0x7f10f80a8ce0 auth_method 0 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 10 monclient(hunting): get_auth_request con 0x7f10f80a8900 auth_method 0 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f77fe640 10 monclient(hunting): get_auth_request con 0x7f10f80a9b80 auth_method 0 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:15.019 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.013+0000 7f10f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.014+0000 7f10f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:15.020 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.014+0000 7f10f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 10 monclient(hunting): handle_auth_done global_id 4240 payload 824 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 1 -- 172.21.15.86:0/2151603106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 msgr2=0x7f10f8146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f8146e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 1 -- 172.21.15.86:0/2151603106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10f80a9b80 msgr2=0x7f10f81d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10f80a9b80 0x7f10f81d76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 10 monclient: _finish_hunting 0 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 1 monclient: found mon.noname-b 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:15.022 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.016+0000 7f10f6ffd640 1 -- 172.21.15.86:0/2151603106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f0009b30 con 0x7f10f80a8900 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.018+0000 7f10f6ffd640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10f80a8900 0x7f10f81427d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f10e800e850 tx=0x7f10e800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.018+0000 7f10f4ff9640 10 client.?.objecter ms_handle_connect 0x7f10f80a8900 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.018+0000 7f10f4ff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 1 -- 172.21.15.86:0/2151603106 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e80095b0 con 0x7f10f80a8900 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: dump: 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:15.023 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: _finish_auth 0 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: _check_auth_tickets 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 1 -- 172.21.15.86:0/2151603106 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e800c940 con 0x7f10f80a8900 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 1 -- 172.21.15.86:0/2151603106 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e800ccd0 con 0x7f10f80a8900 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 5 monclient: authenticate success, global_id 4240 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f4ff9640 10 monclient: dump: 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:15.024 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 10 monclient: _renew_subs 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10f5ffb640 4 set_mon_vals no callback set 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f81481a0 con 0x7f10f80a8900 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 10 monclient: _renew_subs 2023-12-11T20:16:15.025 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:15.026 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f81d7de0 con 0x7f10f80a8900 2023-12-11T20:16:15.026 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.021+0000 7f10fd84b640 1 librados: init done 2023-12-11T20:16:15.026 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.022+0000 7f10fd84b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:15.026 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.022+0000 7f10fd84b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:15.026 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.022+0000 7f10fd84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:15.026 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.022+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f8146ae0 con 0x7f10f80a8900 2023-12-11T20:16:15.039 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.040+0000 7f10f4ff9640 1 -- 172.21.15.86:0/2151603106 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f10e8013470 con 0x7f10f80a8900 2023-12-11T20:16:15.039 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.041+0000 7f10f4ff9640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f10d806c7e0 0x7f10d806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:15.039 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.041+0000 7f10effff640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f10d806c7e0 0x7f10d806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:15.039 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.041+0000 7f10effff640 10 monclient: get_auth_request con 0x7f10d806c7e0 auth_method 0 2023-12-11T20:16:15.040 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.042+0000 7f10f4ff9640 10 client.4240.objecter ms_handle_connect 0x7f10d806c7e0 2023-12-11T20:16:15.040 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.042+0000 7f10effff640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f10d806c7e0 0x7f10d806eca0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f10f81424f0 tx=0x7f10e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:15.041 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.043+0000 7f10f4ff9640 1 -- 172.21.15.86:0/2151603106 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f10e800ce30 con 0x7f10f80a8900 2023-12-11T20:16:15.041 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.043+0000 7f10f4ff9640 10 client.4240.objecter ms_dispatch 0x7f10f8000b90 osd_map(10..10 src has 1..10) v4 2023-12-11T20:16:15.041 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.043+0000 7f10f4ff9640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2023-12-11T20:16:15.041 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.043+0000 7f10f4ff9640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2023-12-11T20:16:15.041 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.043+0000 7f10f4ff9640 20 client.4240.objecter dump_active .. 0 homeless 2023-12-11T20:16:15.095 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:09.966 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.107 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:09.996 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.157 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:10.041 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.224 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:10.114 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.233 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:10.127 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.292 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.294+0000 7f10f4ff9640 1 -- 172.21.15.86:0/2151603106 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10e8054bd0 con 0x7f10f80a8900 2023-12-11T20:16:15.292 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.294+0000 7f10f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:15.292 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.294+0000 7f10f4ff9640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:15.332 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:10.219 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.350 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:10.235 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.353 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:10.250 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.384 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:10.255 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.416 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:10.306 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.445 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:10.329 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.490 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.492+0000 7f10fd84b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:15.490 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.492+0000 7f10fd84b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:15.490 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.492+0000 7f10fd84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:15.490 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.492+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10f8146850 con 0x7f10f80a8900 2023-12-11T20:16:15.505 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.507+0000 7f10f4ff9640 1 -- 172.21.15.86:0/2151603106 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f10e8017090 con 0x7f10f80a8900 2023-12-11T20:16:15.505 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.507+0000 7f10f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:15.505 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.507+0000 7f10f4ff9640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:15.505 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-11T20:16:15.505 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:04.659077+0000","last_up_change":"0.000000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:15.507 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 10 librados: watch_flush enter 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 10 librados: watch_flush exit 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 20 client.4240.objecter shutdown clearing up homeless session... 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 10 client.4240.objecter successfully canceled tick 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f10d806c7e0 msgr2=0x7f10d806eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f10d806c7e0 0x7f10d806eca0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f10f81424f0 tx=0x7f10e0009040 comp rx=0 tx=0).stop 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 10 monclient: shutdown 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10f80a8900 msgr2=0x7f10f81427d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10f80a8900 0x7f10f81427d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f10e800e850 tx=0x7f10e800ed10 comp rx=0 tx=0).stop 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 shutdown_connections 2023-12-11T20:16:15.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f10d806c7e0 0x7f10d806eca0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.509 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10f80a8ce0 0x7f10f8146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.509 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10f80a9b80 0x7f10f81d76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.509 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 --2- 172.21.15.86:0/2151603106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10f80a8900 0x7f10f81427d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:15.509 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 >> 172.21.15.86:0/2151603106 conn(0x7f10f8066f00 msgr2=0x7f10f813eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:15.509 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 shutdown_connections 2023-12-11T20:16:15.509 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 -- 172.21.15.86:0/2151603106 wait complete. 2023-12-11T20:16:15.509 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:15.509+0000 7f10fd84b640 1 librados: shutdown 2023-12-11T20:16:15.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-11T20:16:15.522 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:10.412 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.530 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:10.423 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.601 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:10.488 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.645 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:10.541 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.645 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:10.531 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.678 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:10.549 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.704 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:10.594 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.733 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:10.617 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.814 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:10.704 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.820 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:10.712 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.903 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:10.790 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.933 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:10.818 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.940 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:10.836 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:15.968 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:10.839 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.001 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:10.890 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.033 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:10.917 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.106 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:11.000 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.108 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:10.999 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.198 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:11.085 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.226 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:11.111 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.231 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:11.128 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.261 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:11.132 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.274 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:11.163 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.301 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:11.184 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.370 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:11.264 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.372 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:11.263 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.470 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:11.357 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.494 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:11.379 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.498 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:11.395 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.530 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:11.401 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.538 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:11.427 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.569 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:11.452 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.635 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:11.528 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.642 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:11.532 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.737 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:11.624 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.766 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:11.652 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.776 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:11.673 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.798 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:11.670 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:16.899 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:11.793 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.001 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:11.888 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.013 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:11.902 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.034 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:11.919 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.042 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:11.937 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.066 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:11.950 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.088 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:11.977 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.111 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:12.001 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.121 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:12.005 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.165 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:12.055 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.269 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:12.140 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.367 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:12.239 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.372 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:12.266 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.427 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:12.320 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.512 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:12.408 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.519 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:12.406 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.567 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:12.463 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.572 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:12.457 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.573 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:12.461 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:17.644 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:12.529 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:22.325 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:22.413 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.415+0000 7f94a6bc7640 10 monclient: get_monmap_and_config 2023-12-11T20:16:22.413 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.415+0000 7f94a6bc7640 10 monclient: build_initial_monmap 2023-12-11T20:16:22.413 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.415+0000 7f94a6bc7640 10 monclient: monmap: 2023-12-11T20:16:22.413 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:22.413 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:22.413 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:16:22.415690+0000 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:16:22.415690+0000 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:22.414 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:22.416 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.415+0000 7f94a6bc7640 1 Processor -- start 2023-12-11T20:16:22.416 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.415+0000 7f94a6bc7640 1 -- start start 2023-12-11T20:16:22.416 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.415+0000 7f94a6bc7640 10 monclient: init 2023-12-11T20:16:22.416 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:22.416 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f94a00a8890 0x7f94a00b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 10 monclient(hunting): picked mon.noname-c con 0x7f94a00a8890 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94a00a8c70 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94a013f0a0 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a00b2560 con 0x7f94a013f0a0 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a00a9c30 con 0x7f94a00a8c70 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a00aada0 con 0x7f94a00a8890 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.416+0000 7f94a6bc7640 10 monclient(hunting): authenticate will time out at 1468.745702s 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.417+0000 7f94a4963640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f94a00a8890 0x7f94a00b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.417+0000 7f94a4963640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f94a00a8890 0x7f94a00b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:56386/0 (socket says 172.21.15.86:56386) 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.417+0000 7f94a4963640 1 -- 172.21.15.86:0/1379043335 learned_addr learned my addr 172.21.15.86:0/1379043335 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.417+0000 7f94a4963640 10 monclient(hunting): get_auth_request con 0x7f94a00a8890 auth_method 0 2023-12-11T20:16:22.417 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.417+0000 7f94a4963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:22.418 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.417+0000 7f94a4963640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:22.418 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.417+0000 7f94a4963640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:22.418 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.418+0000 7f94a4963640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:22.418 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.418+0000 7f94a4963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:22.418 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.418+0000 7f94a4963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:22.419 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.420+0000 7f949f7fe640 1 -- 172.21.15.86:0/1379043335 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579604477 0 0) 0x7f94a00b2560 con 0x7f94a013f0a0 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.420+0000 7f949f7fe640 10 monclient(hunting): _negotiate 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.420+0000 7f949f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.420+0000 7f949f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.420+0000 7f949f7fe640 10 monclient(hunting): my global_id is 4191 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.420+0000 7f949f7fe640 1 -- 172.21.15.86:0/1379043335 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9490003520 con 0x7f94a013f0a0 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.421+0000 7f94a4963640 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.421+0000 7f94a4963640 1 -- 172.21.15.86:0/1379043335 >> v1:172.21.15.133:6789/0 conn(0x7f94a00a8c70 legacy=0x7f94a0142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.421+0000 7f94a4963640 1 -- 172.21.15.86:0/1379043335 >> v1:172.21.15.86:6789/0 conn(0x7f94a013f0a0 legacy=0x7f94a013f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.421+0000 7f94a4963640 10 monclient: _finish_hunting 0 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.421+0000 7f94a4963640 1 monclient: found mon.noname-c 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.421+0000 7f94a4963640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.421+0000 7f94a4963640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:22.420 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.421+0000 7f94a4963640 1 -- 172.21.15.86:0/1379043335 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a0142210 con 0x7f94a00a8890 2023-12-11T20:16:22.424 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.426+0000 7f94a4963640 1 --2- 172.21.15.86:0/1379043335 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 0x7f94a00b2020 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9494009870 tx=0x7f949402ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c09f1045136afde server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 1 -- 172.21.15.86:0/1379043335 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949402f850 con 0x7f94a00a8890 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: dump: 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:22.430 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: _finish_auth 0 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: _check_auth_tickets 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f94a6bc7640 5 monclient: authenticate success, global_id 4243 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 1 -- 172.21.15.86:0/1379043335 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949402f9b0 con 0x7f94a00a8890 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f94a6bc7640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 1 -- 172.21.15.86:0/1379043335 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949402fd40 con 0x7f94a00a8890 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f94a6bc7640 4 set_mon_vals no callback set 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:22.431 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.431+0000 7f949f7fe640 10 monclient: dump: 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 10 monclient: shutdown 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 1 -- 172.21.15.86:0/1379043335 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 msgr2=0x7f94a00b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 1 --2- 172.21.15.86:0/1379043335 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 0x7f94a00b2020 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9494009870 tx=0x7f949402ba30 comp rx=0 tx=0).stop 2023-12-11T20:16:22.432 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 1 -- 172.21.15.86:0/1379043335 shutdown_connections 2023-12-11T20:16:22.433 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 1 --2- 172.21.15.86:0/1379043335 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 0x7f94a00b2020 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:22.433 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 1 -- 172.21.15.86:0/1379043335 >> 172.21.15.86:0/1379043335 conn(0x7f94a0066db0 msgr2=0x7f94a00b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:22.433 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 1 -- 172.21.15.86:0/1379043335 shutdown_connections 2023-12-11T20:16:22.433 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 1 -- 172.21.15.86:0/1379043335 wait complete. 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 10 monclient: build_initial_monmap 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.432+0000 7f94a6bc7640 10 monclient: monmap: 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:22.438 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.433+0000 7f94a6bc7640 1 librados: starting msgr at 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.433+0000 7f94a6bc7640 1 librados: starting objecter 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.433+0000 7f94a6bc7640 1 Processor -- start 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.433+0000 7f94a6bc7640 1 -- start start 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.433+0000 7f94a6bc7640 1 librados: setting wanted keys 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.433+0000 7f94a6bc7640 1 librados: calling monclient init 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.433+0000 7f94a6bc7640 10 monclient: init 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 0x7f94a013def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient(hunting): picked mon.noname-b con 0x7f94a00a8890 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:22.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94a00a8c70 0x7f94a01458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient(hunting): picked mon.noname-a con 0x7f94a00a8c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94a013f0a0 0x7f94a0144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient(hunting): picked mon.noname-c con 0x7f94a013f0a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a01451c0 con 0x7f94a00a8c70 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a012c7f0 con 0x7f94a00a8890 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a012cad0 con 0x7f94a013f0a0 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f94a6bc7640 10 monclient(hunting): authenticate will time out at 1468.763571s 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.434+0000 7f949ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94a00a8c70 0x7f94a01458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a5164640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94a013f0a0 0x7f94a0144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:22.440 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f949ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94a00a8c70 0x7f94a01458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54940/0 (socket says 172.21.15.86:54940) 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f949ffff640 1 -- 172.21.15.86:0/374415689 learned_addr learned my addr 172.21.15.86:0/374415689 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f949ffff640 10 monclient(hunting): get_auth_request con 0x7f94a00a8c70 auth_method 0 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f949ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f949ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f949ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a5164640 10 monclient(hunting): get_auth_request con 0x7f94a013f0a0 auth_method 0 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a5164640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a5164640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a5164640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a4963640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 0x7f94a013def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a4963640 10 monclient(hunting): get_auth_request con 0x7f94a00a8890 auth_method 0 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a4963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a4963640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:22.441 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.435+0000 7f94a4963640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.436+0000 7f94a5164640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.436+0000 7f94a5164640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.436+0000 7f94a5164640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.436+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.436+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.436+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a4963640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a4963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a4963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 1 -- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 msgr2=0x7f94a013def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 0x7f94a013def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:22.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 1 -- 172.21.15.86:0/374415689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94a00a8c70 msgr2=0x7f94a01458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:22.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94a00a8c70 0x7f94a01458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:22.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 10 monclient: _finish_hunting 0 2023-12-11T20:16:22.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 1 monclient: found mon.noname-c 2023-12-11T20:16:22.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:22.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:22.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.438+0000 7f94a5164640 1 -- 172.21.15.86:0/374415689 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940094d0 con 0x7f94a013f0a0 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.441+0000 7f94a5164640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94a013f0a0 0x7f94a0144c80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f949800c760 tx=0x7f949800cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.441+0000 7f949d7fa640 10 client.?.objecter ms_handle_connect 0x7f94a013f0a0 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.441+0000 7f949d7fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 1 -- 172.21.15.86:0/374415689 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9498003b70 con 0x7f94a013f0a0 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: dump: 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:22.445 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: _finish_auth 0 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: _check_auth_tickets 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 1 -- 172.21.15.86:0/374415689 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9498003cf0 con 0x7f94a013f0a0 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 1 -- 172.21.15.86:0/374415689 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949800aa90 con 0x7f94a013f0a0 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:22.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949effd640 4 set_mon_vals no callback set 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.445+0000 7f949d7fa640 10 monclient: dump: 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 5 monclient: authenticate success, global_id 4178 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 10 monclient: _renew_subs 2023-12-11T20:16:22.447 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:22.448 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a01453c0 con 0x7f94a013f0a0 2023-12-11T20:16:22.448 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:22.448 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 10 monclient: _renew_subs 2023-12-11T20:16:22.448 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:22.448 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a012d0d0 con 0x7f94a013f0a0 2023-12-11T20:16:22.448 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.446+0000 7f94a6bc7640 1 librados: init done 2023-12-11T20:16:22.449 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.447+0000 7f94a6bc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:22.449 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.447+0000 7f94a6bc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:22.449 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.447+0000 7f94a6bc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:22.449 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.447+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a0072940 con 0x7f94a013f0a0 2023-12-11T20:16:22.562 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:17.451 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:22.590 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:17.472 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:22.692 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.694+0000 7f949d7fa640 1 -- 172.21.15.86:0/374415689 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f949800abf0 con 0x7f94a013f0a0 2023-12-11T20:16:22.693 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.694+0000 7f949d7fa640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f948406c7e0 0x7f948406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:22.693 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.695+0000 7f94a4963640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f948406c7e0 0x7f948406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:22.693 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.695+0000 7f94a4963640 10 monclient: get_auth_request con 0x7f948406c7e0 auth_method 0 2023-12-11T20:16:22.693 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.695+0000 7f949d7fa640 10 client.4178.objecter ms_handle_connect 0x7f948406c7e0 2023-12-11T20:16:22.693 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.695+0000 7f94a4963640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f948406c7e0 0x7f948406eca0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9494036000 tx=0x7f9494008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:22.704 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.706+0000 7f949d7fa640 1 -- 172.21.15.86:0/374415689 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9498058af0 con 0x7f94a013f0a0 2023-12-11T20:16:22.704 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.706+0000 7f949d7fa640 10 client.4178.objecter ms_dispatch 0x7f94a0000b90 osd_map(10..10 src has 1..10) v4 2023-12-11T20:16:22.705 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.706+0000 7f949d7fa640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2023-12-11T20:16:22.705 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.706+0000 7f949d7fa640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2023-12-11T20:16:22.705 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.706+0000 7f949d7fa640 20 client.4178.objecter dump_active .. 0 homeless 2023-12-11T20:16:22.705 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.706+0000 7f949d7fa640 1 -- 172.21.15.86:0/374415689 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9498054e30 con 0x7f94a013f0a0 2023-12-11T20:16:22.705 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.706+0000 7f949d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:22.705 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.706+0000 7f949d7fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:22.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.887+0000 7f94a6bc7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:22.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.887+0000 7f94a6bc7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:22.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.887+0000 7f94a6bc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:22.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.887+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94a004e840 con 0x7f94a013f0a0 2023-12-11T20:16:22.909 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.911+0000 7f949d7fa640 1 -- 172.21.15.86:0/374415689 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9498003e60 con 0x7f94a013f0a0 2023-12-11T20:16:22.910 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.911+0000 7f949d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:22.910 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.911+0000 7f949d7fa640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:22.910 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-11T20:16:22.910 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:04.659077+0000","last_up_change":"0.000000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 10 librados: watch_flush enter 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 10 librados: watch_flush exit 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 20 client.4178.objecter shutdown clearing up homeless session... 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 10 client.4178.objecter successfully canceled tick 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f948406c7e0 msgr2=0x7f948406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f948406c7e0 0x7f948406eca0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9494036000 tx=0x7f9494008030 comp rx=0 tx=0).stop 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 10 monclient: shutdown 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94a013f0a0 msgr2=0x7f94a0144c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:22.912 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.913+0000 7f94a6bc7640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94a013f0a0 0x7f94a0144c80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f949800c760 tx=0x7f949800cc20 comp rx=0 tx=0).stop 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 shutdown_connections 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f948406c7e0 0x7f948406eca0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94a013f0a0 0x7f94a0144c80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94a00a8c70 0x7f94a01458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 --2- 172.21.15.86:0/374415689 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94a00a8890 0x7f94a013def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 >> 172.21.15.86:0/374415689 conn(0x7f94a0066db0 msgr2=0x7f94a0073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 shutdown_connections 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 -- 172.21.15.86:0/374415689 wait complete. 2023-12-11T20:16:22.913 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:22.914+0000 7f94a6bc7640 1 librados: shutdown 2023-12-11T20:16:22.923 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-11T20:16:22.971 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:17.842 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:23.094 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:17.981 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:23.096 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:17.986 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:23.120 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:18.013 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:23.842 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:18.738 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:23.864 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:18.750 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:27.668 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:22.551 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:27.672 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:22.556 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:27.779 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:22.668 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:27.786 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:22.675 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:28.500 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:23.390 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:28.504 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:23.394 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:28.940 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:23.812 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:28.944 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:23.816 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.068 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:23.961 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.071 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:23.965 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.104 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:23.989 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.108 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:23.993 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.255 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:24.142 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.259 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:24.146 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.730 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:29.739 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:24.636 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.745 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:24.642 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:29.818 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.820+0000 7f3a75c9b640 10 monclient: get_monmap_and_config 2023-12-11T20:16:29.818 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.820+0000 7f3a75c9b640 10 monclient: build_initial_monmap 2023-12-11T20:16:29.818 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.820+0000 7f3a75c9b640 10 monclient: monmap: 2023-12-11T20:16:29.818 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:29.818 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:16:29.820712+0000 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:16:29.820712+0000 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:29.819 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 1 Processor -- start 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 1 -- start start 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient: init 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f3a700a8890 0x7f3a700b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient(hunting): picked mon.noname-c con 0x7f3a700a8890 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a700a8c70 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a7013f0a0 0x7f3a7013f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:29.821 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient(hunting): picked mon.noname-b con 0x7f3a7013f0a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7013fa70 con 0x7f3a700a8c70 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a700b2560 con 0x7f3a700a8890 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700a9c30 con 0x7f3a7013f0a0 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.821+0000 7f3a75c9b640 10 monclient(hunting): authenticate will time out at 1476.150921s 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.822+0000 7f3a6effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3a700a8c70 0x7f3a70142730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58510/0 (socket says 172.21.15.86:58510) 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.822+0000 7f3a6ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a7013f0a0 0x7f3a7013f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.822+0000 7f3a6effd640 1 -- 172.21.15.86:0/3368352749 learned_addr learned my addr 172.21.15.86:0/3368352749 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6f7fe640 1 --2- 172.21.15.86:0/3368352749 >> v2:172.21.15.133:3300/0 conn(0x7f3a700a8890 0x7f3a700b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6f7fe640 10 monclient(hunting): get_auth_request con 0x7f3a700a8890 auth_method 0 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:29.822 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:29.824 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6ffff640 10 monclient(hunting): get_auth_request con 0x7f3a7013f0a0 auth_method 0 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.823+0000 7f3a6ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.824+0000 7f3a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.824+0000 7f3a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.824+0000 7f3a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.824+0000 7f3a6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.824+0000 7f3a6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.824+0000 7f3a6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.825+0000 7f3a6e7fc640 1 -- 172.21.15.86:0/3368352749 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300554121 0 0) 0x7f3a7013fa70 con 0x7f3a700a8c70 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.825+0000 7f3a6e7fc640 10 monclient(hunting): _negotiate 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.825+0000 7f3a6e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:29.825 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.825+0000 7f3a6e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.825+0000 7f3a6e7fc640 10 monclient(hunting): my global_id is 4197 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.825+0000 7f3a6e7fc640 1 -- 172.21.15.86:0/3368352749 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a58003520 con 0x7f3a700a8c70 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 1 -- 172.21.15.86:0/3368352749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a7013f0a0 msgr2=0x7f3a7013f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 1 --2- 172.21.15.86:0/3368352749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a7013f0a0 0x7f3a7013f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 1 -- 172.21.15.86:0/3368352749 >> v1:172.21.15.86:6789/0 conn(0x7f3a700a8c70 legacy=0x7f3a70142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 10 monclient: _finish_hunting 0 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 1 monclient: found mon.noname-c 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:29.826 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.826+0000 7f3a6f7fe640 1 -- 172.21.15.86:0/3368352749 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a700aada0 con 0x7f3a700a8890 2023-12-11T20:16:29.827 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.829+0000 7f3a6f7fe640 1 --2- 172.21.15.86:0/3368352749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8890 0x7f3a700b2020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3a600097a0 tx=0x7f3a6002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62eae1c849704985 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:29.830 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a6e7fc640 1 -- 172.21.15.86:0/3368352749 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60031830 con 0x7f3a700a8890 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a6e7fc640 10 monclient: dump: 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a6e7fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:29.831 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a6e7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:29.832 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:29.832 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a75c9b640 5 monclient: authenticate success, global_id 4249 2023-12-11T20:16:29.832 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.832+0000 7f3a75c9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.834+0000 7f3a6e7fc640 1 -- 172.21.15.86:0/3368352749 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a60031990 con 0x7f3a700a8890 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.834+0000 7f3a6e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.834+0000 7f3a6e7fc640 1 -- 172.21.15.86:0/3368352749 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60031d40 con 0x7f3a700a8890 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.834+0000 7f3a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.834+0000 7f3a6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.834+0000 7f3a6e7fc640 10 monclient: dump: 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:29.836 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.834+0000 7f3a75c9b640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.834+0000 7f3a75c9b640 4 set_mon_vals no callback set 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 10 monclient: shutdown 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 1 -- 172.21.15.86:0/3368352749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8890 msgr2=0x7f3a700b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/3368352749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8890 0x7f3a700b2020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3a600097a0 tx=0x7f3a6002b8d0 comp rx=0 tx=0).stop 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 1 -- 172.21.15.86:0/3368352749 shutdown_connections 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/3368352749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a7013f0a0 0x7f3a7013f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:29.837 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/3368352749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8890 0x7f3a700b2020 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 1 -- 172.21.15.86:0/3368352749 >> 172.21.15.86:0/3368352749 conn(0x7f3a70066db0 msgr2=0x7f3a700b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 1 -- 172.21.15.86:0/3368352749 shutdown_connections 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.835+0000 7f3a75c9b640 1 -- 172.21.15.86:0/3368352749 wait complete. 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.836+0000 7f3a75c9b640 10 monclient: build_initial_monmap 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.836+0000 7f3a75c9b640 10 monclient: monmap: 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:29.838 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.836+0000 7f3a75c9b640 1 librados: starting msgr at 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.836+0000 7f3a75c9b640 1 librados: starting objecter 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.836+0000 7f3a75c9b640 1 Processor -- start 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 1 -- start start 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 1 librados: setting wanted keys 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 1 librados: calling monclient init 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 10 monclient: init 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a700a8890 0x7f3a701443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 10 monclient(hunting): picked mon.noname-c con 0x7f3a700a8890 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.837+0000 7f3a75c9b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8c70 0x7f3a701439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.838+0000 7f3a75c9b640 10 monclient(hunting): picked mon.noname-b con 0x7f3a700a8c70 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:29.839 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.838+0000 7f3a75c9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a7013f0a0 0x7f3a701ded00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:29.843 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.838+0000 7f3a6f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a700a8890 0x7f3a701443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:29.843 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.839+0000 7f3a6f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a700a8890 0x7f3a701443d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:60400/0 (socket says 172.21.15.86:60400) 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.839+0000 7f3a6effd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8c70 0x7f3a701439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.839+0000 7f3a75c9b640 10 monclient(hunting): picked mon.noname-a con 0x7f3a7013f0a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.839+0000 7f3a75c9b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.839+0000 7f3a6f7fe640 1 -- 172.21.15.86:0/2378138466 learned_addr learned my addr 172.21.15.86:0/2378138466 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.839+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70145e50 con 0x7f3a7013f0a0 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.839+0000 7f3a6ffff640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a7013f0a0 0x7f3a701ded00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.839+0000 7f3a75c9b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70143f30 con 0x7f3a700a8c70 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a75c9b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a701de7e0 con 0x7f3a700a8890 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a75c9b640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:29.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a75c9b640 10 monclient(hunting): authenticate will time out at 1476.169402s 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6f7fe640 10 monclient(hunting): get_auth_request con 0x7f3a700a8890 auth_method 0 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6ffff640 10 monclient(hunting): get_auth_request con 0x7f3a7013f0a0 auth_method 0 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.840+0000 7f3a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6effd640 10 monclient(hunting): get_auth_request con 0x7f3a700a8c70 auth_method 0 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:29.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6effd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.841+0000 7f3a6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.844+0000 7f3a6effd640 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.844+0000 7f3a6effd640 1 -- 172.21.15.86:0/2378138466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a700a8890 msgr2=0x7f3a701443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.844+0000 7f3a6effd640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a700a8890 0x7f3a701443d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:29.846 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.844+0000 7f3a6effd640 1 -- 172.21.15.86:0/2378138466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a7013f0a0 msgr2=0x7f3a701ded00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:29.847 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.847+0000 7f3a6effd640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a7013f0a0 0x7f3a701ded00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.847+0000 7f3a6effd640 10 monclient: _finish_hunting 0 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.847+0000 7f3a6effd640 1 monclient: found mon.noname-b 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.847+0000 7f3a6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.847+0000 7f3a6effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.847+0000 7f3a6effd640 1 -- 172.21.15.86:0/2378138466 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a60009400 con 0x7f3a700a8c70 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.847+0000 7f3a6ffff640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a7013f0a0 0x7f3a701ded00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.848+0000 7f3a6effd640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8c70 0x7f3a701439f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400c3e0 tx=0x7f3a6400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.848+0000 7f3a57fff640 10 client.?.objecter ms_handle_connect 0x7f3a700a8c70 2023-12-11T20:16:29.848 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.848+0000 7f3a57fff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:29.853 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 1 -- 172.21.15.86:0/2378138466 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a64005ce0 con 0x7f3a700a8c70 2023-12-11T20:16:29.853 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:29.853 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 10 monclient: dump: 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 10 monclient: _finish_auth 0 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:29.854 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 1 -- 172.21.15.86:0/2378138466 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a64005e40 con 0x7f3a700a8c70 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a57fff640 1 -- 172.21.15.86:0/2378138466 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a640072d0 con 0x7f3a700a8c70 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a75c9b640 5 monclient: authenticate success, global_id 4252 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a75c9b640 10 monclient: _renew_subs 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a75c9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a70146050 con 0x7f3a700a8c70 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.853+0000 7f3a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.852+0000 7f3a6dffb640 4 set_mon_vals no callback set 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.853+0000 7f3a57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.853+0000 7f3a57fff640 10 monclient: dump: 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:29.855 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.853+0000 7f3a75c9b640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.853+0000 7f3a75c9b640 10 monclient: _renew_subs 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.853+0000 7f3a75c9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.853+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a70135990 con 0x7f3a700a8c70 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.853+0000 7f3a75c9b640 1 librados: init done 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.854+0000 7f3a75c9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.854+0000 7f3a75c9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.854+0000 7f3a75c9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:29.856 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:29.854+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a701de640 con 0x7f3a700a8c70 2023-12-11T20:16:30.166 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.168+0000 7f3a57fff640 1 -- 172.21.15.86:0/2378138466 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f3a64007430 con 0x7f3a700a8c70 2023-12-11T20:16:30.167 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.169+0000 7f3a57fff640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3a5c06c810 0x7f3a5c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:30.167 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.169+0000 7f3a6f7fe640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3a5c06c810 0x7f3a5c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:30.167 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.169+0000 7f3a6f7fe640 10 monclient: get_auth_request con 0x7f3a5c06c810 auth_method 0 2023-12-11T20:16:30.167 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.169+0000 7f3a57fff640 10 client.4252.objecter ms_handle_connect 0x7f3a5c06c810 2023-12-11T20:16:30.167 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.169+0000 7f3a6f7fe640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3a5c06c810 0x7f3a5c06ecd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a6002b910 tx=0x7f3a6003d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:30.189 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.190+0000 7f3a57fff640 1 -- 172.21.15.86:0/2378138466 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f3a64057fb0 con 0x7f3a700a8c70 2023-12-11T20:16:30.189 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.190+0000 7f3a57fff640 10 client.4252.objecter ms_dispatch 0x7f3a70000b90 osd_map(10..10 src has 1..10) v4 2023-12-11T20:16:30.189 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.190+0000 7f3a57fff640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2023-12-11T20:16:30.189 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.190+0000 7f3a57fff640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2023-12-11T20:16:30.189 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.190+0000 7f3a57fff640 20 client.4252.objecter dump_active .. 0 homeless 2023-12-11T20:16:30.189 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.190+0000 7f3a57fff640 1 -- 172.21.15.86:0/2378138466 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a640542f0 con 0x7f3a700a8c70 2023-12-11T20:16:30.189 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.191+0000 7f3a57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:30.189 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.191+0000 7f3a57fff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:30.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.470+0000 7f3a75c9b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:30.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.470+0000 7f3a75c9b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:30.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.470+0000 7f3a75c9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:30.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.470+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a701de640 con 0x7f3a700a8c70 2023-12-11T20:16:30.485 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.487+0000 7f3a57fff640 1 -- 172.21.15.86:0/2378138466 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3a701de640 con 0x7f3a700a8c70 2023-12-11T20:16:30.485 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.487+0000 7f3a57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:30.485 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.487+0000 7f3a57fff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:30.485 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-11T20:16:30.486 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:04.659077+0000","last_up_change":"0.000000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 10 librados: watch_flush enter 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 10 librados: watch_flush exit 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 20 client.4252.objecter shutdown clearing up homeless session... 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 10 client.4252.objecter successfully canceled tick 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3a5c06c810 msgr2=0x7f3a5c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3a5c06c810 0x7f3a5c06ecd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a6002b910 tx=0x7f3a6003d040 comp rx=0 tx=0).stop 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 10 monclient: shutdown 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8c70 msgr2=0x7f3a701439f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8c70 0x7f3a701439f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400c3e0 tx=0x7f3a6400c8a0 comp rx=0 tx=0).stop 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 shutdown_connections 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3a5c06c810 0x7f3a5c06ecd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:30.488 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a7013f0a0 0x7f3a701ded00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:30.489 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a700a8c70 0x7f3a701439f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:30.489 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 --2- 172.21.15.86:0/2378138466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a700a8890 0x7f3a701443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:30.489 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 >> 172.21.15.86:0/2378138466 conn(0x7f3a70066db0 msgr2=0x7f3a70132ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:30.489 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 shutdown_connections 2023-12-11T20:16:30.489 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 -- 172.21.15.86:0/2378138466 wait complete. 2023-12-11T20:16:30.489 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:30.489+0000 7f3a75c9b640 1 librados: shutdown 2023-12-11T20:16:30.498 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-11T20:16:31.590 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:26.474 86788== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:31.734 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:26.623 86786== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:32.843 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:27.733 86785== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:33.460 INFO:tasks.ceph.osd.1.smithi086.stderr:==00:00:00:28.353 86844== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:33.563 INFO:tasks.ceph.osd.2.smithi086.stderr:==00:00:00:28.450 86845== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:33.569 INFO:tasks.ceph.osd.3.smithi086.stderr:==00:00:00:28.441 86846== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:33.655 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:28.540 86787== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:33.942 INFO:tasks.ceph.osd.0.smithi086.stderr:==00:00:00:28.838 86843== Warning: unimplemented fcntl command: 1038 2023-12-11T20:16:37.305 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:37.438 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.439+0000 7f2e2c79e640 10 monclient: get_monmap_and_config 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.439+0000 7f2e2c79e640 10 monclient: build_initial_monmap 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.439+0000 7f2e2c79e640 10 monclient: monmap: 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:16:37.441046+0000 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:16:37.441046+0000 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:37.439 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.440+0000 7f2e2c79e640 1 Processor -- start 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.440+0000 7f2e2c79e640 1 -- start start 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.440+0000 7f2e2c79e640 10 monclient: init 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e240a8900 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f2e240a9b80 0x7f2e2413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient(hunting): picked mon.noname-c con 0x7f2e240a9b80 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:37.442 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e240a8ce0 0x7f2e241453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient(hunting): picked mon.noname-a con 0x7f2e240a8ce0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240b1b80 con 0x7f2e240a8900 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e240aa870 con 0x7f2e240a8ce0 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e240aab50 con 0x7f2e240a9b80 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.441+0000 7f2e2c79e640 10 monclient(hunting): authenticate will time out at 1483.772006s 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.442+0000 7f2e29d39640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f2e240a9b80 0x7f2e2413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.442+0000 7f2e29d39640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f2e240a9b80 0x7f2e2413f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:53600/0 (socket says 172.21.15.86:53600) 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.442+0000 7f2e29d39640 1 -- 172.21.15.86:0/167983145 learned_addr learned my addr 172.21.15.86:0/167983145 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.442+0000 7f2e29d39640 10 monclient(hunting): get_auth_request con 0x7f2e240a9b80 auth_method 0 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.442+0000 7f2e29d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:37.443 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.442+0000 7f2e29d39640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:37.444 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.442+0000 7f2e29d39640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:37.444 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.443+0000 7f2e2ad3b640 1 --2- 172.21.15.86:0/167983145 >> v2:172.21.15.86:3300/0 conn(0x7f2e240a8ce0 0x7f2e241453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.443+0000 7f2e2ad3b640 10 monclient(hunting): get_auth_request con 0x7f2e240a8ce0 auth_method 0 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.443+0000 7f2e2ad3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.443+0000 7f2e2ad3b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.443+0000 7f2e2ad3b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.443+0000 7f2e29d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.443+0000 7f2e29d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.443+0000 7f2e29d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.444+0000 7f2e2ad3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.444+0000 7f2e2ad3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.444+0000 7f2e2ad3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:37.445 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 10 monclient(hunting): handle_auth_done global_id 4255 payload 274 2023-12-11T20:16:37.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 1 -- 172.21.15.86:0/167983145 >> v2:172.21.15.86:3300/0 conn(0x7f2e240a8ce0 msgr2=0x7f2e241453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:37.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 1 --2- 172.21.15.86:0/167983145 >> v2:172.21.15.86:3300/0 conn(0x7f2e240a8ce0 0x7f2e241453f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:37.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 1 -- 172.21.15.86:0/167983145 >> v1:172.21.15.86:6789/0 conn(0x7f2e240a8900 legacy=0x7f2e2413efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:37.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 10 monclient: _finish_hunting 0 2023-12-11T20:16:37.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 1 monclient: found mon.noname-c 2023-12-11T20:16:37.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:37.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:37.446 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.446+0000 7f2e29d39640 1 -- 172.21.15.86:0/167983145 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24145930 con 0x7f2e240a9b80 2023-12-11T20:16:37.448 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.449+0000 7f2e29d39640 1 --2- 172.21.15.86:0/167983145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 0x7f2e2413f6e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e240b38b0 tx=0x7f2e1402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72a744bebc895a10 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.453+0000 7f2e29538640 1 -- 172.21.15.86:0/167983145 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e14030860 con 0x7f2e240a9b80 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.453+0000 7f2e29538640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.453+0000 7f2e29538640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.453+0000 7f2e29538640 10 monclient: dump: 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:37.453 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.453+0000 7f2e29538640 10 monclient: _finish_auth 0 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.453+0000 7f2e29538640 10 monclient: _check_auth_tickets 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.453+0000 7f2e29538640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e29538640 1 -- 172.21.15.86:0/167983145 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e140309c0 con 0x7f2e240a9b80 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e29538640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e29538640 1 -- 172.21.15.86:0/167983145 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e14030d50 con 0x7f2e240a9b80 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e29538640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e29538640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:37.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e29538640 10 monclient: dump: 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e2c79e640 5 monclient: authenticate success, global_id 4255 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e2c79e640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:37.455 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e2c79e640 4 set_mon_vals no callback set 2023-12-11T20:16:37.461 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e2c79e640 10 monclient: shutdown 2023-12-11T20:16:37.461 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e2c79e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:37.461 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e2c79e640 1 -- 172.21.15.86:0/167983145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 msgr2=0x7f2e2413f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:37.461 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.454+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/167983145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 0x7f2e2413f6e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e240b38b0 tx=0x7f2e1402b9a0 comp rx=0 tx=0).stop 2023-12-11T20:16:37.461 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.455+0000 7f2e2c79e640 1 -- 172.21.15.86:0/167983145 shutdown_connections 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.455+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/167983145 >> v2:172.21.15.86:3300/0 conn(0x7f2e240a8ce0 0x7f2e241453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.455+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/167983145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 0x7f2e2413f6e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.455+0000 7f2e2c79e640 1 -- 172.21.15.86:0/167983145 >> 172.21.15.86:0/167983145 conn(0x7f2e24066f00 msgr2=0x7f2e240b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.455+0000 7f2e2c79e640 1 -- 172.21.15.86:0/167983145 shutdown_connections 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.455+0000 7f2e2c79e640 1 -- 172.21.15.86:0/167983145 wait complete. 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 10 monclient: build_initial_monmap 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 10 monclient: monmap: 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:37.462 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 1 librados: starting msgr at 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 1 librados: starting objecter 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 1 Processor -- start 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 1 -- start start 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 1 librados: setting wanted keys 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 1 librados: calling monclient init 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.456+0000 7f2e2c79e640 10 monclient: init 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.457+0000 7f2e2c79e640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.457+0000 7f2e2c79e640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.457+0000 7f2e2c79e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e240a8900 0x7f2e24146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.457+0000 7f2e2c79e640 10 monclient(hunting): picked mon.noname-a con 0x7f2e240a8900 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:37.463 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.457+0000 7f2e2c79e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e240a8ce0 0x7f2e240b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:37.464 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 10 monclient(hunting): picked mon.noname-c con 0x7f2e240a8ce0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:37.464 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e29d39640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e240a8ce0 0x7f2e240b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:37.464 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2a53a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e240a8900 0x7f2e24146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:37.464 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 0x7f2e240b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:37.464 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 10 monclient(hunting): picked mon.noname-b con 0x7f2e240a9b80 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:37.464 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:37.464 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e240b7b90 con 0x7f2e240a8900 2023-12-11T20:16:37.464 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e29d39640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e240a8ce0 0x7f2e240b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54750/0 (socket says 172.21.15.86:54750) 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e29d39640 1 -- 172.21.15.86:0/1241655293 learned_addr learned my addr 172.21.15.86:0/1241655293 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e240b8020 con 0x7f2e240a9b80 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e240b5d70 con 0x7f2e240a8ce0 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.458+0000 7f2e2c79e640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2c79e640 10 monclient(hunting): authenticate will time out at 1483.789164s 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e29d39640 10 monclient(hunting): get_auth_request con 0x7f2e240a8ce0 auth_method 0 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e29d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e29d39640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e29d39640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2a53a640 10 monclient(hunting): get_auth_request con 0x7f2e240a8900 auth_method 0 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2a53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:37.465 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2a53a640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2a53a640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2ad3b640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 0x7f2e240b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2a53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2a53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2a53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2ad3b640 10 monclient(hunting): get_auth_request con 0x7f2e240a9b80 auth_method 0 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2ad3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2ad3b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e2ad3b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e29d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e29d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.459+0000 7f2e29d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.460+0000 7f2e2ad3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:37.466 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.460+0000 7f2e2ad3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:37.467 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.460+0000 7f2e2ad3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:37.468 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 10 monclient(hunting): handle_auth_done global_id 4209 payload 824 2023-12-11T20:16:37.468 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 1 -- 172.21.15.86:0/1241655293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e240a8ce0 msgr2=0x7f2e240b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:37.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e240a8ce0 0x7f2e240b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:37.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 1 -- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 msgr2=0x7f2e240b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:37.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 0x7f2e240b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:37.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 10 monclient: _finish_hunting 0 2023-12-11T20:16:37.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 1 monclient: found mon.noname-a 2023-12-11T20:16:37.469 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:37.470 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:37.470 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.463+0000 7f2e2a53a640 1 -- 172.21.15.86:0/1241655293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140094d0 con 0x7f2e240a8900 2023-12-11T20:16:37.470 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.467+0000 7f2e2a53a640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e240a8900 0x7f2e24146930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000c390 tx=0x7f2e2000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:37.470 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.468+0000 7f2e1affd640 10 client.?.objecter ms_handle_connect 0x7f2e240a8900 2023-12-11T20:16:37.470 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.468+0000 7f2e1affd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:37.473 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.470+0000 7f2e1affd640 1 -- 172.21.15.86:0/1241655293 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e200052d0 con 0x7f2e240a8900 2023-12-11T20:16:37.473 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.470+0000 7f2e1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:37.473 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.470+0000 7f2e1affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:37.474 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.470+0000 7f2e1affd640 10 monclient: dump: 2023-12-11T20:16:37.474 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:37.474 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:37.474 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:37.474 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:37.474 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:37.474 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:37.475 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:37.475 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:37.475 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:37.475 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:37.475 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.470+0000 7f2e1affd640 10 monclient: _finish_auth 0 2023-12-11T20:16:37.475 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.470+0000 7f2e1affd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:37.476 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:37.476 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 5 monclient: authenticate success, global_id 4209 2023-12-11T20:16:37.476 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 10 monclient: _renew_subs 2023-12-11T20:16:37.476 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:37.476 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e24147d90 con 0x7f2e240a8900 2023-12-11T20:16:37.476 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:37.476 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 10 monclient: _renew_subs 2023-12-11T20:16:37.477 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:37.477 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e24146e70 con 0x7f2e240a8900 2023-12-11T20:16:37.477 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.471+0000 7f2e2c79e640 1 librados: init done 2023-12-11T20:16:37.477 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.472+0000 7f2e2c79e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:37.477 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.472+0000 7f2e2c79e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:37.477 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.472+0000 7f2e2c79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:37.477 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.472+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e240b3010 con 0x7f2e240a8900 2023-12-11T20:16:37.478 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.473+0000 7f2e1affd640 1 -- 172.21.15.86:0/1241655293 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e20005430 con 0x7f2e240a8900 2023-12-11T20:16:37.478 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.473+0000 7f2e1affd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:37.478 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.473+0000 7f2e1affd640 1 -- 172.21.15.86:0/1241655293 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20007250 con 0x7f2e240a8900 2023-12-11T20:16:37.478 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.473+0000 7f2e28d37640 4 set_mon_vals no callback set 2023-12-11T20:16:37.478 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.473+0000 7f2e1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:37.478 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.473+0000 7f2e1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:37.479 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.473+0000 7f2e1affd640 10 monclient: dump: 2023-12-11T20:16:37.479 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:37.479 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:37.479 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:37.479 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:37.479 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:37.479 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:37.480 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:37.480 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:37.480 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:37.480 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:37.841 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.842+0000 7f2e1affd640 1 -- 172.21.15.86:0/1241655293 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f2e200073b0 con 0x7f2e240a8900 2023-12-11T20:16:37.841 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.842+0000 7f2e1affd640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f2e0806c7c0 0x7f2e0806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:37.842 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.842+0000 7f2e29d39640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f2e0806c7c0 0x7f2e0806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:37.842 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.843+0000 7f2e29d39640 10 monclient: get_auth_request con 0x7f2e0806c7c0 auth_method 0 2023-12-11T20:16:37.842 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.843+0000 7f2e1affd640 10 client.4209.objecter ms_handle_connect 0x7f2e0806c7c0 2023-12-11T20:16:37.842 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.843+0000 7f2e29d39640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f2e0806c7c0 0x7f2e0806ec80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2e240b79f0 tx=0x7f2e1403b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:37.865 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.866+0000 7f2e1affd640 1 -- 172.21.15.86:0/1241655293 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2e20007660 con 0x7f2e240a8900 2023-12-11T20:16:37.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.866+0000 7f2e1affd640 10 client.4209.objecter ms_dispatch 0x7f2e24000b90 osd_map(10..10 src has 1..10) v4 2023-12-11T20:16:37.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.866+0000 7f2e1affd640 3 client.4209.objecter handle_osd_map got epochs [10,10] > 0 2023-12-11T20:16:37.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.866+0000 7f2e1affd640 3 client.4209.objecter handle_osd_map decoding full epoch 10 2023-12-11T20:16:37.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.866+0000 7f2e1affd640 20 client.4209.objecter dump_active .. 0 homeless 2023-12-11T20:16:37.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.866+0000 7f2e1affd640 1 -- 172.21.15.86:0/1241655293 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e20054050 con 0x7f2e240a8900 2023-12-11T20:16:37.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.866+0000 7f2e1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:37.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:37.866+0000 7f2e1affd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:38.090 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.090+0000 7f2e2c79e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:38.090 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.090+0000 7f2e2c79e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:38.090 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.090+0000 7f2e2c79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:38.090 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.090+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e2406b320 con 0x7f2e240a8900 2023-12-11T20:16:38.158 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.159+0000 7f2e1affd640 1 -- 172.21.15.86:0/1241655293 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2e20017020 con 0x7f2e240a8900 2023-12-11T20:16:38.159 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.159+0000 7f2e1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:38.159 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.159+0000 7f2e1affd640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:38.159 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-11T20:16:38.159 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:04.659077+0000","last_up_change":"0.000000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 10 librados: watch_flush enter 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 10 librados: watch_flush exit 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 20 client.4209.objecter shutdown clearing up homeless session... 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 10 client.4209.objecter successfully canceled tick 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f2e0806c7c0 msgr2=0x7f2e0806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f2e0806c7c0 0x7f2e0806ec80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2e240b79f0 tx=0x7f2e1403b040 comp rx=0 tx=0).stop 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 10 monclient: shutdown 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e240a8900 msgr2=0x7f2e24146930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e240a8900 0x7f2e24146930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000c390 tx=0x7f2e2000c850 comp rx=0 tx=0).stop 2023-12-11T20:16:38.162 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 shutdown_connections 2023-12-11T20:16:38.163 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f2e0806c7c0 0x7f2e0806ec80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:38.163 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e240a8ce0 0x7f2e240b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:38.163 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e240a9b80 0x7f2e240b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:38.163 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 --2- 172.21.15.86:0/1241655293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e240a8900 0x7f2e24146930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:38.163 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.162+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 >> 172.21.15.86:0/1241655293 conn(0x7f2e24066f00 msgr2=0x7f2e2413aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:38.163 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.163+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 shutdown_connections 2023-12-11T20:16:38.163 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.163+0000 7f2e2c79e640 1 -- 172.21.15.86:0/1241655293 wait complete. 2023-12-11T20:16:38.163 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:38.163+0000 7f2e2c79e640 1 librados: shutdown 2023-12-11T20:16:38.176 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-11T20:16:40.049 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:16:40.048+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-12-11T20:16:40.367 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:16:40.366+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-12-11T20:16:41.272 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:16:41.271+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-12-11T20:16:41.884 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:16:41.880+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-12-11T20:16:42.025 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:16:42.024+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-12-11T20:16:42.082 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:16:42.081+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-12-11T20:16:42.259 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:16:42.257+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-12-11T20:16:42.440 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:16:42.438+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-12-11T20:16:43.104 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:16:43.101+0000 ac7d640 -1 osd.7 0 waiting for initial osdmap 2023-12-11T20:16:43.161 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:16:43.159+0000 c493640 -1 osd.5 0 waiting for initial osdmap 2023-12-11T20:16:43.540 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:16:43.528+0000 12257640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-11T20:16:43.569 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:16:43.557+0000 12257640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-11T20:16:44.163 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:16:44.160+0000 c493640 -1 osd.4 0 waiting for initial osdmap 2023-12-11T20:16:44.577 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:16:44.564+0000 12257640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-11T20:16:44.982 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:45.102 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.103+0000 7f38fb5e7640 10 monclient: get_monmap_and_config 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.103+0000 7f38fb5e7640 10 monclient: build_initial_monmap 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.103+0000 7f38fb5e7640 10 monclient: monmap: 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:16:45.104781+0000 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:16:45.104781+0000 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:45.103 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:45.104 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:45.104 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:45.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.104+0000 7f38fb5e7640 1 Processor -- start 2023-12-11T20:16:45.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.104+0000 7f38fb5e7640 1 -- start start 2023-12-11T20:16:45.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.104+0000 7f38fb5e7640 10 monclient: init 2023-12-11T20:16:45.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:45.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-12-11T20:16:45.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f38f40a8900 0x7f38f413efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:45.107 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 10 monclient(hunting): picked mon.noname-c con 0x7f38f40a8900 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38f40a9b80 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a8ce0 0x7f38f41453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 10 monclient(hunting): picked mon.noname-b con 0x7f38f40a8ce0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f40b1b80 con 0x7f38f40a9b80 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f40aa870 con 0x7f38f40a8900 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f40aab50 con 0x7f38f40a8ce0 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.105+0000 7f38fb5e7640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.106+0000 7f38fb5e7640 10 monclient(hunting): authenticate will time out at 1491.436136s 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.106+0000 7f38f9b84640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a8ce0 0x7f38f41453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.106+0000 7f38f9b84640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a8ce0 0x7f38f41453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54776/0 (socket says 172.21.15.86:54776) 2023-12-11T20:16:45.108 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.106+0000 7f38f9b84640 1 -- 172.21.15.86:0/3465591677 learned_addr learned my addr 172.21.15.86:0/3465591677 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:45.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.106+0000 7f38f9b84640 10 monclient(hunting): get_auth_request con 0x7f38f40a8ce0 auth_method 0 2023-12-11T20:16:45.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.106+0000 7f38f9b84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:45.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.106+0000 7f38f9b84640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:45.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.106+0000 7f38f9b84640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:45.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.107+0000 7f38f9b84640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:45.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.107+0000 7f38f9b84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:45.109 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.107+0000 7f38f9b84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:45.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2023-12-11T20:16:45.110 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 1 -- 172.21.15.86:0/3465591677 >> v2:172.21.15.133:3300/0 conn(0x7f38f40a8900 msgr2=0x7f38f413efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:45.111 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 1 --2- 172.21.15.86:0/3465591677 >> v2:172.21.15.133:3300/0 conn(0x7f38f40a8900 0x7f38f413efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.111 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 1 -- 172.21.15.86:0/3465591677 >> v1:172.21.15.86:6789/0 conn(0x7f38f40a9b80 legacy=0x7f38f413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:45.111 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 10 monclient: _finish_hunting 0 2023-12-11T20:16:45.111 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 1 monclient: found mon.noname-b 2023-12-11T20:16:45.111 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:45.111 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:45.111 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.109+0000 7f38f9b84640 1 -- 172.21.15.86:0/3465591677 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4145930 con 0x7f38f40a8ce0 2023-12-11T20:16:45.111 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.111+0000 7f38f9b84640 1 --2- 172.21.15.86:0/3465591677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a8ce0 0x7f38f41453f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009e50 tx=0x7f38ec02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=886e0aefbf6a0f37 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:45.117 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 1 -- 172.21.15.86:0/3465591677 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec02ca80 con 0x7f38f40a8ce0 2023-12-11T20:16:45.117 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:45.117 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:45.117 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: dump: 2023-12-11T20:16:45.117 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:45.117 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:45.117 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: _finish_auth 0 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 1 -- 172.21.15.86:0/3465591677 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec02cbe0 con 0x7f38f40a8ce0 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38fb5e7640 5 monclient: authenticate success, global_id 4214 2023-12-11T20:16:45.118 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38fb5e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38fb5e7640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38fb5e7640 4 set_mon_vals no callback set 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 1 -- 172.21.15.86:0/3465591677 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec0343f0 con 0x7f38f40a8ce0 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.115+0000 7f38ebfff640 10 monclient: dump: 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:45.119 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.116+0000 7f38fb5e7640 10 monclient: shutdown 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.116+0000 7f38fb5e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.116+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3465591677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a8ce0 msgr2=0x7f38f41453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.116+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3465591677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a8ce0 0x7f38f41453f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009e50 tx=0x7f38ec02bee0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.117+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3465591677 shutdown_connections 2023-12-11T20:16:45.120 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.117+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3465591677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a8ce0 0x7f38f41453f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.121 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.117+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3465591677 >> v2:172.21.15.133:3300/0 conn(0x7f38f40a8900 0x7f38f413efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.121 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.117+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3465591677 >> 172.21.15.86:0/3465591677 conn(0x7f38f4066f00 msgr2=0x7f38f40b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:45.121 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.117+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3465591677 shutdown_connections 2023-12-11T20:16:45.121 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.118+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3465591677 wait complete. 2023-12-11T20:16:45.123 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.119+0000 7f38fb5e7640 10 monclient: build_initial_monmap 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.119+0000 7f38fb5e7640 10 monclient: monmap: 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.120+0000 7f38fb5e7640 1 librados: starting msgr at 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.120+0000 7f38fb5e7640 1 librados: starting objecter 2023-12-11T20:16:45.124 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.120+0000 7f38fb5e7640 1 Processor -- start 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.120+0000 7f38fb5e7640 1 -- start start 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.120+0000 7f38fb5e7640 1 librados: setting wanted keys 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.120+0000 7f38fb5e7640 1 librados: calling monclient init 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.120+0000 7f38fb5e7640 10 monclient: init 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.121+0000 7f38fb5e7640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.121+0000 7f38fb5e7640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.121+0000 7f38fb5e7640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38f40a8900 0x7f38f4146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.121+0000 7f38fb5e7640 10 monclient(hunting): picked mon.noname-b con 0x7f38f40a8900 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.121+0000 7f38fb5e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38f40a8ce0 0x7f38f41d77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 10 monclient(hunting): picked mon.noname-a con 0x7f38f40a8ce0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a9b80 0x7f38f41d7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 10 monclient(hunting): picked mon.noname-c con 0x7f38f40a9b80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f41482c0 con 0x7f38f40a8ce0 2023-12-11T20:16:45.125 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:45.126 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f41484a0 con 0x7f38f40a8900 2023-12-11T20:16:45.126 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:45.126 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4148780 con 0x7f38f40a9b80 2023-12-11T20:16:45.126 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:45.126 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.122+0000 7f38fb5e7640 10 monclient(hunting): authenticate will time out at 1491.452799s 2023-12-11T20:16:45.153 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.153+0000 7f38f8b82640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38f40a8ce0 0x7f38f41d77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:45.155 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.156+0000 7f38f8b82640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38f40a8ce0 0x7f38f41d77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57594/0 (socket says 172.21.15.86:57594) 2023-12-11T20:16:45.155 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.156+0000 7f38f8b82640 1 -- 172.21.15.86:0/3545856176 learned_addr learned my addr 172.21.15.86:0/3545856176 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:45.155 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.156+0000 7f38f8b82640 10 monclient(hunting): get_auth_request con 0x7f38f40a8ce0 auth_method 0 2023-12-11T20:16:45.155 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.156+0000 7f38f8b82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:45.155 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.156+0000 7f38f8b82640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:45.155 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.156+0000 7f38f8b82640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:45.157 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.158+0000 7f38f8b82640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:45.157 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.158+0000 7f38f8b82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:45.157 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.158+0000 7f38f8b82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:45.165 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2023-12-11T20:16:45.165 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 1 -- 172.21.15.86:0/3545856176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a9b80 msgr2=0x7f38f41d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:45.165 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a9b80 0x7f38f41d7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.165 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 1 -- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38f40a8900 msgr2=0x7f38f4146930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:45.166 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38f40a8900 0x7f38f4146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.166 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 10 monclient: _finish_hunting 0 2023-12-11T20:16:45.166 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 1 monclient: found mon.noname-a 2023-12-11T20:16:45.166 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:45.166 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:45.166 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.166+0000 7f38f8b82640 1 -- 172.21.15.86:0/3545856176 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec009b30 con 0x7f38f40a8ce0 2023-12-11T20:16:45.170 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:16:45.168+0000 c493640 -1 osd.2 0 waiting for initial osdmap 2023-12-11T20:16:45.173 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.174+0000 7f38f8b82640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38f40a8ce0 0x7f38f41d77c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f38e400b3c0 tx=0x7f38e400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:45.173 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.174+0000 7f38e9ffb640 10 client.?.objecter ms_handle_connect 0x7f38f40a8ce0 2023-12-11T20:16:45.173 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.174+0000 7f38e9ffb640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:45.177 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:16:45.175+0000 ac7d640 -1 osd.3 0 waiting for initial osdmap 2023-12-11T20:16:45.179 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 1 -- 172.21.15.86:0/3545856176 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4011830 con 0x7f38f40a8ce0 2023-12-11T20:16:45.179 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: dump: 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: _finish_auth 0 2023-12-11T20:16:45.180 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 1 -- 172.21.15.86:0/3545856176 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e4011990 con 0x7f38f40a8ce0 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38fb5e7640 5 monclient: authenticate success, global_id 4233 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38fb5e7640 10 monclient: _renew_subs 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38fb5e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f4147030 con 0x7f38f40a8ce0 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 1 -- 172.21.15.86:0/3545856176 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4011d60 con 0x7f38f40a8ce0 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38e9ffb640 10 monclient: dump: 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:45.181 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:45.182 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:45.182 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:45.182 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:45.182 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:45.182 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:45.182 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:45.182 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.180+0000 7f38eb7fe640 4 set_mon_vals no callback set 2023-12-11T20:16:45.182 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.181+0000 7f38fb5e7640 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:45.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.181+0000 7f38fb5e7640 10 monclient: _renew_subs 2023-12-11T20:16:45.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.181+0000 7f38fb5e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:45.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.181+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f40b80f0 con 0x7f38f40a8ce0 2023-12-11T20:16:45.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.182+0000 7f38fb5e7640 1 librados: init done 2023-12-11T20:16:45.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.183+0000 7f38fb5e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:45.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.183+0000 7f38fb5e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:45.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.183+0000 7f38fb5e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:45.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.183+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f4147270 con 0x7f38f40a8ce0 2023-12-11T20:16:45.315 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:16:45.313+0000 c493640 -1 osd.0 0 waiting for initial osdmap 2023-12-11T20:16:45.329 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:16:45.326+0000 c493640 -1 osd.6 0 waiting for initial osdmap 2023-12-11T20:16:45.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.508+0000 7f38e9ffb640 1 -- 172.21.15.86:0/3545856176 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f38e401b460 con 0x7f38f40a8ce0 2023-12-11T20:16:45.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.508+0000 7f38e9ffb640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f38d806c790 0x7f38d806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:45.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.509+0000 7f38f9383640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f38d806c790 0x7f38d806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:45.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.509+0000 7f38f9383640 10 monclient: get_auth_request con 0x7f38d806c790 auth_method 0 2023-12-11T20:16:45.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.509+0000 7f38f9383640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f38d806c790 0x7f38d806ec50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009710 tx=0x7f38f0006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:45.508 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.509+0000 7f38e9ffb640 10 client.4233.objecter ms_handle_connect 0x7f38d806c790 2023-12-11T20:16:45.532 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.533+0000 7f38e9ffb640 1 -- 172.21.15.86:0/3545856176 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3071+0+0 (secure 0 0 0) 0x7f38e40591f0 con 0x7f38f40a8ce0 2023-12-11T20:16:45.532 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.533+0000 7f38e9ffb640 10 client.4233.objecter ms_dispatch 0x7f38f4000b90 osd_map(14..14 src has 1..14) v4 2023-12-11T20:16:45.532 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.533+0000 7f38e9ffb640 3 client.4233.objecter handle_osd_map got epochs [14,14] > 0 2023-12-11T20:16:45.533 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.533+0000 7f38e9ffb640 3 client.4233.objecter handle_osd_map decoding full epoch 14 2023-12-11T20:16:45.533 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.533+0000 7f38e9ffb640 20 client.4233.objecter dump_active .. 0 homeless 2023-12-11T20:16:45.533 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.533+0000 7f38e9ffb640 1 -- 172.21.15.86:0/3545856176 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38e4054e30 con 0x7f38f40a8ce0 2023-12-11T20:16:45.533 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.533+0000 7f38e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:45.533 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.533+0000 7f38e9ffb640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:45.681 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:16:45.663+0000 11a56640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-11T20:16:45.699 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:16:45.698+0000 ac7d640 -1 osd.1 0 waiting for initial osdmap 2023-12-11T20:16:45.715 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:16:45.702+0000 11a56640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-11T20:16:45.756 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:16:45.743+0000 12257640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-11T20:16:45.809 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.810+0000 7f38fb5e7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:45.809 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.810+0000 7f38fb5e7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:45.809 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.810+0000 7f38fb5e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:45.809 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.810+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38f4146e70 con 0x7f38f40a8ce0 2023-12-11T20:16:45.840 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.841+0000 7f38e9ffb640 1 -- 172.21.15.86:0/3545856176 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8561 (secure 0 0 0) 0x7f38e4058450 con 0x7f38f40a8ce0 2023-12-11T20:16:45.840 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.841+0000 7f38e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:45.840 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.841+0000 7f38e9ffb640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:45.841 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-11T20:16:45.841 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":14,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:44.803034+0000","last_up_change":"2023-12-11T20:16:44.803034+0000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6819","nonce":1433273204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6821","nonce":1433273204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6825","nonce":1433273204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6823","nonce":1433273204}]},"public_addr":"172.21.15.133:6819/1433273204","cluster_addr":"172.21.15.133:6821/1433273204","heartbeat_back_addr":"172.21.15.133:6825/1433273204","heartbeat_front_addr":"172.21.15.133:6823/1433273204","state":["exists","up"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6803","nonce":3582621102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6805","nonce":3582621102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6809","nonce":3582621102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6807","nonce":3582621102}]},"public_addr":"172.21.15.133:6803/3582621102","cluster_addr":"172.21.15.133:6805/3582621102","heartbeat_back_addr":"172.21.15.133:6809/3582621102","heartbeat_front_addr":"172.21.15.133:6807/3582621102","state":["exists","up"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6811","nonce":1136721322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6813","nonce":1136721322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6817","nonce":1136721322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6815","nonce":1136721322}]},"public_addr":"172.21.15.133:6811/1136721322","cluster_addr":"172.21.15.133:6813/1136721322","heartbeat_back_addr":"172.21.15.133:6817/1136721322","heartbeat_front_addr":"172.21.15.133:6815/1136721322","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:42.326615+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:45.843 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.843+0000 7f38fb5e7640 10 librados: watch_flush enter 2023-12-11T20:16:45.843 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.843+0000 7f38fb5e7640 10 librados: watch_flush exit 2023-12-11T20:16:45.843 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.843+0000 7f38fb5e7640 20 client.4233.objecter shutdown clearing up homeless session... 2023-12-11T20:16:45.843 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.843+0000 7f38fb5e7640 10 client.4233.objecter successfully canceled tick 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f38d806c790 msgr2=0x7f38d806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f38d806c790 0x7f38d806ec50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009710 tx=0x7f38f0006c90 comp rx=0 tx=0).stop 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 10 monclient: shutdown 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38f40a8ce0 msgr2=0x7f38f41d77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38f40a8ce0 0x7f38f41d77c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f38e400b3c0 tx=0x7f38e400b880 comp rx=0 tx=0).stop 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 shutdown_connections 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f38d806c790 0x7f38d806ec50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38f40a9b80 0x7f38f41d7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38f40a8ce0 0x7f38f41d77c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 --2- 172.21.15.86:0/3545856176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38f40a8900 0x7f38f4146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 >> 172.21.15.86:0/3545856176 conn(0x7f38f4066f00 msgr2=0x7f38f413eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 shutdown_connections 2023-12-11T20:16:45.844 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 -- 172.21.15.86:0/3545856176 wait complete. 2023-12-11T20:16:45.845 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:45.844+0000 7f38fb5e7640 1 librados: shutdown 2023-12-11T20:16:45.849 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:16:45.834+0000 11a56640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-11T20:16:45.856 DEBUG:teuthology.misc:3 of 8 OSDs are up 2023-12-11T20:16:46.086 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:16:46.073+0000 11a56640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-11T20:16:52.663 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.764+0000 7fa584a89640 10 monclient: get_monmap_and_config 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.764+0000 7fa584a89640 10 monclient: build_initial_monmap 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.765+0000 7fa584a89640 10 monclient: monmap: 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:16:52.766520+0000 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:16:52.766520+0000 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:52.764 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:52.765 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:52.765 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:52.765 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:52.765 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.765+0000 7fa584a89640 1 Processor -- start 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.765+0000 7fa584a89640 1 -- start start 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.765+0000 7fa584a89640 10 monclient: init 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5800a8890 0x7fa5800b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient(hunting): picked mon.noname-a con 0x7fa5800a8890 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5800a8c70 0x7fa580142730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient(hunting): picked mon.noname-b con 0x7fa5800a8c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:52.766 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa58013f0a0 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:52.767 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5800b2560 con 0x7fa58013f0a0 2023-12-11T20:16:52.767 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:52.767 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5800a9c30 con 0x7fa5800a8890 2023-12-11T20:16:52.767 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:52.767 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5800aada0 con 0x7fa5800a8c70 2023-12-11T20:16:52.767 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:52.767 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.766+0000 7fa584a89640 10 monclient(hunting): authenticate will time out at 1499.096615s 2023-12-11T20:16:52.861 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.862+0000 7fa57e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5800a8890 0x7fa5800b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:52.863 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.865+0000 7fa57e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5800a8890 0x7fa5800b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59476/0 (socket says 172.21.15.86:59476) 2023-12-11T20:16:52.864 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.865+0000 7fa57e59c640 1 -- 172.21.15.86:0/193280628 learned_addr learned my addr 172.21.15.86:0/193280628 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:52.864 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.865+0000 7fa57e59c640 10 monclient(hunting): get_auth_request con 0x7fa5800a8890 auth_method 0 2023-12-11T20:16:52.864 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.865+0000 7fa57e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:52.864 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.865+0000 7fa57e59c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:52.864 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.865+0000 7fa57e59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:52.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.867+0000 7fa57e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:52.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.867+0000 7fa57e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:52.866 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.867+0000 7fa57e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:52.868 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2023-12-11T20:16:52.868 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 1 -- 172.21.15.86:0/193280628 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5800a8c70 msgr2=0x7fa580142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:52.869 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 1 --2- 172.21.15.86:0/193280628 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5800a8c70 0x7fa580142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:52.869 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 1 -- 172.21.15.86:0/193280628 >> v1:172.21.15.86:6789/0 conn(0x7fa58013f0a0 legacy=0x7fa580145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:52.869 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 10 monclient: _finish_hunting 0 2023-12-11T20:16:52.869 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 1 monclient: found mon.noname-a 2023-12-11T20:16:52.869 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:52.869 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:52.869 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.869+0000 7fa57e59c640 1 -- 172.21.15.86:0/193280628 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa580142210 con 0x7fa5800a8890 2023-12-11T20:16:52.873 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.874+0000 7fa57e59c640 1 --2- 172.21.15.86:0/193280628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8890 0x7fa5800b2020 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa570009870 tx=0x7fa57002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acab5b06991f61af server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:52.878 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 1 -- 172.21.15.86:0/193280628 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570030840 con 0x7fa5800a8890 2023-12-11T20:16:52.878 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:52.878 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:52.878 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: dump: 2023-12-11T20:16:52.878 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:52.878 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:52.878 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:52.878 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: _finish_auth 0 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: _check_auth_tickets 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 1 -- 172.21.15.86:0/193280628 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5700309a0 con 0x7fa5800a8890 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa584a89640 5 monclient: authenticate success, global_id 4242 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa584a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:52.879 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa584a89640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa584a89640 4 set_mon_vals no callback set 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 1 -- 172.21.15.86:0/193280628 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570030d30 con 0x7fa5800a8890 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa57d59a640 10 monclient: dump: 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:52.880 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:52.881 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:52.883 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa584a89640 10 monclient: shutdown 2023-12-11T20:16:52.883 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa584a89640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:52.883 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa584a89640 1 -- 172.21.15.86:0/193280628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8890 msgr2=0x7fa5800b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:52.883 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.879+0000 7fa584a89640 1 --2- 172.21.15.86:0/193280628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8890 0x7fa5800b2020 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa570009870 tx=0x7fa57002b980 comp rx=0 tx=0).stop 2023-12-11T20:16:52.883 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.880+0000 7fa584a89640 1 -- 172.21.15.86:0/193280628 shutdown_connections 2023-12-11T20:16:52.883 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.880+0000 7fa584a89640 1 --2- 172.21.15.86:0/193280628 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5800a8c70 0x7fa580142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.880+0000 7fa584a89640 1 --2- 172.21.15.86:0/193280628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8890 0x7fa5800b2020 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.880+0000 7fa584a89640 1 -- 172.21.15.86:0/193280628 >> 172.21.15.86:0/193280628 conn(0x7fa580066db0 msgr2=0x7fa5800b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.880+0000 7fa584a89640 1 -- 172.21.15.86:0/193280628 shutdown_connections 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.880+0000 7fa584a89640 1 -- 172.21.15.86:0/193280628 wait complete. 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.880+0000 7fa584a89640 10 monclient: build_initial_monmap 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.880+0000 7fa584a89640 10 monclient: monmap: 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:52.884 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.881+0000 7fa584a89640 1 librados: starting msgr at 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.881+0000 7fa584a89640 1 librados: starting objecter 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.881+0000 7fa584a89640 1 Processor -- start 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.881+0000 7fa584a89640 1 -- start start 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.881+0000 7fa584a89640 1 librados: setting wanted keys 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.881+0000 7fa584a89640 1 librados: calling monclient init 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.881+0000 7fa584a89640 10 monclient: init 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5800a8890 0x7fa5801ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient(hunting): picked mon.noname-c con 0x7fa5800a8890 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8c70 0x7fa5801d7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:52.885 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient(hunting): picked mon.noname-a con 0x7fa5800a8c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa58013f0a0 0x7fa58013a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient(hunting): picked mon.noname-b con 0x7fa58013f0a0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5801d7810 con 0x7fa5800a8c70 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5800abc70 con 0x7fa58013f0a0 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5801cd870 con 0x7fa5800a8890 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.882+0000 7fa584a89640 10 monclient(hunting): authenticate will time out at 1499.212374s 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.883+0000 7fa57dd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8c70 0x7fa5801d7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.883+0000 7fa57dd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8c70 0x7fa5801d7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59480/0 (socket says 172.21.15.86:59480) 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.883+0000 7fa57dd9b640 1 -- 172.21.15.86:0/3088925538 learned_addr learned my addr 172.21.15.86:0/3088925538 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:52.886 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.883+0000 7fa57dd9b640 10 monclient(hunting): get_auth_request con 0x7fa5800a8c70 auth_method 0 2023-12-11T20:16:52.887 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.883+0000 7fa57dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:52.887 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.883+0000 7fa57dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:52.887 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.883+0000 7fa57dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:52.887 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.884+0000 7fa57dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:52.887 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.884+0000 7fa57dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:52.887 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.884+0000 7fa57dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 1 -- 172.21.15.86:0/3088925538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5800a8890 msgr2=0x7fa5801ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5800a8890 0x7fa5801ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 1 -- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa58013f0a0 msgr2=0x7fa58013a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa58013f0a0 0x7fa58013a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 10 monclient: _finish_hunting 0 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 1 monclient: found mon.noname-a 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:52.888 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.888+0000 7fa57dd9b640 1 -- 172.21.15.86:0/3088925538 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5700094d0 con 0x7fa5800a8c70 2023-12-11T20:16:52.890 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.891+0000 7fa57dd9b640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8c70 0x7fa5801d7c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa57400b4d0 tx=0x7fa57400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:52.890 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.891+0000 7fa56effd640 10 client.?.objecter ms_handle_connect 0x7fa5800a8c70 2023-12-11T20:16:52.890 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.891+0000 7fa56effd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 1 -- 172.21.15.86:0/3088925538 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574010870 con 0x7fa5800a8c70 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: dump: 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:52.895 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: _finish_auth 0 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 1 -- 172.21.15.86:0/3088925538 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5740109d0 con 0x7fa5800a8c70 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 5 monclient: authenticate success, global_id 4245 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 1 -- 172.21.15.86:0/3088925538 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574010d80 con 0x7fa5800a8c70 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 10 monclient: _renew_subs 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa57cd99640 4 set_mon_vals no callback set 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5801ced40 con 0x7fa5800a8c70 2023-12-11T20:16:52.896 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa56effd640 10 monclient: dump: 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr: 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 10 monclient: _renew_subs 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:52.897 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5801ce2c0 con 0x7fa5800a8c70 2023-12-11T20:16:52.898 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.895+0000 7fa584a89640 1 librados: init done 2023-12-11T20:16:52.898 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.896+0000 7fa584a89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:52.898 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.896+0000 7fa584a89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:52.898 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.896+0000 7fa584a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:52.898 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.896+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5801d7570 con 0x7fa5800a8c70 2023-12-11T20:16:52.917 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.918+0000 7fa56effd640 1 -- 172.21.15.86:0/3088925538 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa574023480 con 0x7fa5800a8c70 2023-12-11T20:16:52.918 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.919+0000 7fa56effd640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa56006c870 0x7fa56006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:52.918 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.919+0000 7fa57e59c640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa56006c870 0x7fa56006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:52.918 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.919+0000 7fa57e59c640 10 monclient: get_auth_request con 0x7fa56006c870 auth_method 0 2023-12-11T20:16:52.919 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.920+0000 7fa56effd640 10 client.4245.objecter ms_handle_connect 0x7fa56006c870 2023-12-11T20:16:52.919 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:52.920+0000 7fa57e59c640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa56006c870 0x7fa56006ed30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa570036000 tx=0x7fa57003b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:53.182 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.182+0000 7fa56effd640 1 -- 172.21.15.86:0/3088925538 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa574058b70 con 0x7fa5800a8c70 2023-12-11T20:16:53.182 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.182+0000 7fa56effd640 10 client.4245.objecter ms_dispatch 0x7fa580000b90 osd_map(18..18 src has 1..18) v4 2023-12-11T20:16:53.182 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.182+0000 7fa56effd640 3 client.4245.objecter handle_osd_map got epochs [18,18] > 0 2023-12-11T20:16:53.182 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.182+0000 7fa56effd640 3 client.4245.objecter handle_osd_map decoding full epoch 18 2023-12-11T20:16:53.182 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.183+0000 7fa56effd640 20 client.4245.objecter dump_active .. 0 homeless 2023-12-11T20:16:53.182 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.183+0000 7fa56effd640 1 -- 172.21.15.86:0/3088925538 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5740547b0 con 0x7fa5800a8c70 2023-12-11T20:16:53.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.183+0000 7fa56effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:53.183 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.183+0000 7fa56effd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:53.364 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.365+0000 7fa584a89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:53.365 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.365+0000 7fa584a89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:53.365 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.365+0000 7fa584a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:53.365 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.365+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5801d7950 con 0x7fa5800a8c70 2023-12-11T20:16:53.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.455+0000 7fa56effd640 1 -- 172.21.15.86:0/3088925538 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12851 (secure 0 0 0) 0x7fa5801d7950 con 0x7fa5800a8c70 2023-12-11T20:16:53.454 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.455+0000 7fa56effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:53.455 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.455+0000 7fa56effd640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:53.455 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-11T20:16:53.455 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":18,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:49.944626+0000","last_up_change":"2023-12-11T20:16:47.001015+0000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-11T20:16:44.927047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6821","nonce":1914311689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6826","nonce":1914311689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6831","nonce":1914311689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6829","nonce":1914311689}]},"public_addr":"172.21.15.86:6821/1914311689","cluster_addr":"172.21.15.86:6826/1914311689","heartbeat_back_addr":"172.21.15.86:6831/1914311689","heartbeat_front_addr":"172.21.15.86:6829/1914311689","state":["exists","up"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6809","nonce":4202858331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6811","nonce":4202858331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6815","nonce":4202858331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6813","nonce":4202858331}]},"public_addr":"172.21.15.86:6809/4202858331","cluster_addr":"172.21.15.86:6811/4202858331","heartbeat_back_addr":"172.21.15.86:6815/4202858331","heartbeat_front_addr":"172.21.15.86:6813/4202858331","state":["exists","up"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6817","nonce":406445159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6819","nonce":406445159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6825","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6827","nonce":406445159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6823","nonce":406445159}]},"public_addr":"172.21.15.86:6817/406445159","cluster_addr":"172.21.15.86:6819/406445159","heartbeat_back_addr":"172.21.15.86:6827/406445159","heartbeat_front_addr":"172.21.15.86:6823/406445159","state":["exists","up"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6801","nonce":4064445038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6803","nonce":4064445038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6807","nonce":4064445038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6805","nonce":4064445038}]},"public_addr":"172.21.15.86:6801/4064445038","cluster_addr":"172.21.15.86:6803/4064445038","heartbeat_back_addr":"172.21.15.86:6807/4064445038","heartbeat_front_addr":"172.21.15.86:6805/4064445038","state":["exists","up"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6819","nonce":1433273204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6821","nonce":1433273204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6825","nonce":1433273204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6823","nonce":1433273204}]},"public_addr":"172.21.15.133:6819/1433273204","cluster_addr":"172.21.15.133:6821/1433273204","heartbeat_back_addr":"172.21.15.133:6825/1433273204","heartbeat_front_addr":"172.21.15.133:6823/1433273204","state":["exists","up"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6803","nonce":3582621102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6805","nonce":3582621102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6809","nonce":3582621102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6807","nonce":3582621102}]},"public_addr":"172.21.15.133:6803/3582621102","cluster_addr":"172.21.15.133:6805/3582621102","heartbeat_back_addr":"172.21.15.133:6809/3582621102","heartbeat_front_addr":"172.21.15.133:6807/3582621102","state":["exists","up"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6827","nonce":1611390427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6829","nonce":1611390427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6833","nonce":1611390427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6831","nonce":1611390427}]},"public_addr":"172.21.15.133:6827/1611390427","cluster_addr":"172.21.15.133:6829/1611390427","heartbeat_back_addr":"172.21.15.133:6833/1611390427","heartbeat_front_addr":"172.21.15.133:6831/1611390427","state":["exists","up"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6811","nonce":1136721322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6813","nonce":1136721322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6817","nonce":1136721322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6815","nonce":1136721322}]},"public_addr":"172.21.15.133:6811/1136721322","cluster_addr":"172.21.15.133:6813/1136721322","heartbeat_back_addr":"172.21.15.133:6817/1136721322","heartbeat_front_addr":"172.21.15.133:6815/1136721322","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.469087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.889798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.985199+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.354101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.245493+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:41.942057+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.009608+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:42.326615+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:53.457 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 10 librados: watch_flush enter 2023-12-11T20:16:53.457 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 10 librados: watch_flush exit 2023-12-11T20:16:53.457 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 20 client.4245.objecter shutdown clearing up homeless session... 2023-12-11T20:16:53.457 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 10 client.4245.objecter successfully canceled tick 2023-12-11T20:16:53.457 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa56006c870 msgr2=0x7fa56006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:53.457 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa56006c870 0x7fa56006ed30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa570036000 tx=0x7fa57003b040 comp rx=0 tx=0).stop 2023-12-11T20:16:53.457 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 10 monclient: shutdown 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8c70 msgr2=0x7fa5801d7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8c70 0x7fa5801d7c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa57400b4d0 tx=0x7fa57400b990 comp rx=0 tx=0).stop 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 shutdown_connections 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa56006c870 0x7fa56006ed30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa58013f0a0 0x7fa58013a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5800a8c70 0x7fa5801d7c30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 --2- 172.21.15.86:0/3088925538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5800a8890 0x7fa5801ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 >> 172.21.15.86:0/3088925538 conn(0x7fa580066db0 msgr2=0x7fa5800b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 shutdown_connections 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 -- 172.21.15.86:0/3088925538 wait complete. 2023-12-11T20:16:53.458 INFO:teuthology.misc.health.smithi086.stderr:2023-12-11T20:16:53.457+0000 7fa584a89640 1 librados: shutdown 2023-12-11T20:16:53.466 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-11T20:16:53.466 INFO:tasks.ceph:Creating RBD pool 2023-12-11T20:16:53.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-11T20:16:53.566 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.567+0000 7f880494e640 10 monclient: get_monmap_and_config 2023-12-11T20:16:53.566 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.567+0000 7f880494e640 10 monclient: build_initial_monmap 2023-12-11T20:16:53.566 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.567+0000 7f880494e640 10 monclient: monmap: 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:53.568750+0000 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:53.568750+0000 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.567+0000 7f880494e640 1 Processor -- start 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.567+0000 7f880494e640 1 -- start start 2023-12-11T20:16:53.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.567+0000 7f880494e640 10 monclient: init 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f88000a8930 0x7f88000a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient(hunting): picked mon.noname-c con 0x7f88000a8930 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88000b21a0 0x7f88001424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient(hunting): picked mon.noname-a con 0x7f88000b21a0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88001429f0 0x7f880013f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient(hunting): picked mon.noname-b con 0x7f88001429f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88000aa980 con 0x7f88000b21a0 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88000aabf0 con 0x7f88000a8930 2023-12-11T20:16:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88000aaed0 con 0x7f88001429f0 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.568+0000 7f880494e640 10 monclient(hunting): authenticate will time out at 1499.898749s 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88001429f0 0x7f880013f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fe59c640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f88000a8930 0x7f88000a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fdd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88000b21a0 0x7f88001424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fe59c640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f88000a8930 0x7f88000a8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:33336/0 (socket says 172.21.15.86:33336) 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fe59c640 1 -- 172.21.15.86:0/1602340186 learned_addr learned my addr 172.21.15.86:0/1602340186 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fe59c640 10 monclient(hunting): get_auth_request con 0x7f88000a8930 auth_method 0 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fe59c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fe59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:53.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fed9d640 10 monclient(hunting): get_auth_request con 0x7f88001429f0 auth_method 0 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fed9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fdd9b640 10 monclient(hunting): get_auth_request con 0x7f88000b21a0 auth_method 0 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fdd9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.569+0000 7f87fdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:53.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.570+0000 7f87fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 1 -- 172.21.15.86:0/1602340186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88001429f0 msgr2=0x7f880013f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 1 --2- 172.21.15.86:0/1602340186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88001429f0 0x7f880013f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 1 -- 172.21.15.86:0/1602340186 >> v2:172.21.15.86:3300/0 conn(0x7f88000b21a0 msgr2=0x7f88001424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 1 --2- 172.21.15.86:0/1602340186 >> v2:172.21.15.86:3300/0 conn(0x7f88000b21a0 0x7f88001424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 10 monclient: _finish_hunting 0 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 1 monclient: found mon.noname-c 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:53.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.571+0000 7f87fe59c640 1 -- 172.21.15.86:0/1602340186 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8800141f90 con 0x7f88000a8930 2023-12-11T20:16:53.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.573+0000 7f87fe59c640 1 --2- 172.21.15.86:0/1602340186 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88000a8930 0x7f88000a8d10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009870 tx=0x7f87f002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6bb12254c5fdc17 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:53.576 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 1 -- 172.21.15.86:0/1602340186 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f00317c0 con 0x7f88000a8930 2023-12-11T20:16:53.576 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:53.576 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:53.576 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: dump: 2023-12-11T20:16:53.576 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:53.576 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:53.576 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: _finish_auth 0 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: _check_auth_tickets 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 1 -- 172.21.15.86:0/1602340186 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f0031920 con 0x7f88000a8930 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f880494e640 5 monclient: authenticate success, global_id 4291 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f880494e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:53.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f880494e640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f880494e640 4 set_mon_vals no callback set 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 1 -- 172.21.15.86:0/1602340186 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0031cb0 con 0x7f88000a8930 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.577+0000 7f87fd59a640 10 monclient: dump: 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:53.578 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:53.579 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 10 monclient: shutdown 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 -- 172.21.15.86:0/1602340186 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88000a8930 msgr2=0x7f88000a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 --2- 172.21.15.86:0/1602340186 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88000a8930 0x7f88000a8d10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009870 tx=0x7f87f002b860 comp rx=0 tx=0).stop 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 -- 172.21.15.86:0/1602340186 shutdown_connections 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 --2- 172.21.15.86:0/1602340186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88001429f0 0x7f880013f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 --2- 172.21.15.86:0/1602340186 >> v2:172.21.15.86:3300/0 conn(0x7f88000b21a0 0x7f88001424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 --2- 172.21.15.86:0/1602340186 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88000a8930 0x7f88000a8d10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 -- 172.21.15.86:0/1602340186 >> 172.21.15.86:0/1602340186 conn(0x7f88000b82f0 msgr2=0x7f88000b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 -- 172.21.15.86:0/1602340186 shutdown_connections 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.578+0000 7f880494e640 1 -- 172.21.15.86:0/1602340186 wait complete. 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 10 monclient: build_initial_monmap 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 10 monclient: monmap: 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:53.583 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 1 librados: starting msgr at 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 1 librados: starting objecter 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 1 Processor -- start 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 1 -- start start 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 1 librados: setting wanted keys 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 1 librados: calling monclient init 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.579+0000 7f880494e640 10 monclient: init 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88000a8930 0x7f8800144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient(hunting): picked mon.noname-c con 0x7f88000a8930 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88000b21a0 0x7f88001d6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient(hunting): picked mon.noname-a con 0x7f88000b21a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88001429f0 0x7f88001d72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient(hunting): picked mon.noname-b con 0x7f88001429f0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88000ab840 con 0x7f88000b21a0 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88000abd60 con 0x7f88001429f0 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8800145120 con 0x7f88000a8930 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f880494e640 10 monclient(hunting): authenticate will time out at 1499.910269s 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fed9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88001429f0 0x7f88001d72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fe59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88000a8930 0x7f8800144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fdd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88000b21a0 0x7f88001d6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fed9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88001429f0 0x7f88001d72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:33340/0 (socket says 172.21.15.86:33340) 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fed9d640 1 -- 172.21.15.86:0/4144868844 learned_addr learned my addr 172.21.15.86:0/4144868844 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:53.585 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fed9d640 10 monclient(hunting): get_auth_request con 0x7f88001429f0 auth_method 0 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fed9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.580+0000 7f87fed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fe59c640 10 monclient(hunting): get_auth_request con 0x7f88000a8930 auth_method 0 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fe59c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fe59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fdd9b640 10 monclient(hunting): get_auth_request con 0x7f88000b21a0 auth_method 0 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fdd9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:53.587 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:53.587 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:53.587 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:53.587 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:53.587 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.581+0000 7f87fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:53.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2023-12-11T20:16:53.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 1 -- 172.21.15.86:0/4144868844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88000a8930 msgr2=0x7f8800144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:53.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88000a8930 0x7f8800144b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 1 -- 172.21.15.86:0/4144868844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88000b21a0 msgr2=0x7f88001d6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:53.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88000b21a0 0x7f88001d6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:53.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 10 monclient: _finish_hunting 0 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 1 monclient: found mon.noname-b 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.584+0000 7f87fed9d640 1 -- 172.21.15.86:0/4144868844 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f00094d0 con 0x7f88001429f0 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.586+0000 7f87fed9d640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88001429f0 0x7f88001d72f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f87f800efc0 tx=0x7f87f800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.586+0000 7f87eeffd640 10 client.?.objecter ms_handle_connect 0x7f88001429f0 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.586+0000 7f87eeffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 1 -- 172.21.15.86:0/4144868844 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8009ba0 con 0x7f88001429f0 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: dump: 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:53.590 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: _finish_auth 0 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 1 -- 172.21.15.86:0/4144868844 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f800f040 con 0x7f88001429f0 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 5 monclient: authenticate success, global_id 4294 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 10 monclient: _renew_subs 2023-12-11T20:16:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88001d7830 con 0x7f88001429f0 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 1 -- 172.21.15.86:0/4144868844 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f80146c0 con 0x7f88001429f0 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87eeffd640 10 monclient: dump: 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:53.592 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f87fcd99640 4 set_mon_vals no callback set 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 10 monclient: _renew_subs 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88000b6f10 con 0x7f88001429f0 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.589+0000 7f880494e640 1 librados: init done 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.590+0000 7f880494e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.590+0000 7f880494e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.590+0000 7f880494e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:53.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.590+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88000aba80 con 0x7f88001429f0 2023-12-11T20:16:53.827 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.828+0000 7f87eeffd640 1 -- 172.21.15.86:0/4144868844 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f87f8014820 con 0x7f88001429f0 2023-12-11T20:16:53.827 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.828+0000 7f87eeffd640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f87e006c820 0x7f87e006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:53.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.829+0000 7f87fe59c640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f87e006c820 0x7f87e006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:53.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.829+0000 7f87fe59c640 10 monclient: get_auth_request con 0x7f87e006c820 auth_method 0 2023-12-11T20:16:53.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.829+0000 7f87eeffd640 10 client.4294.objecter ms_handle_connect 0x7f87e006c820 2023-12-11T20:16:53.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.829+0000 7f87fe59c640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f87e006c820 0x7f87e006ece0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f87f0036000 tx=0x7f87f003d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:53.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.831+0000 7f87eeffd640 1 -- 172.21.15.86:0/4144868844 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f87f8059400 con 0x7f88001429f0 2023-12-11T20:16:53.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.831+0000 7f87eeffd640 10 client.4294.objecter ms_dispatch 0x7f8800000b90 osd_map(18..18 src has 1..18) v4 2023-12-11T20:16:53.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.831+0000 7f87eeffd640 3 client.4294.objecter handle_osd_map got epochs [18,18] > 0 2023-12-11T20:16:53.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.831+0000 7f87eeffd640 3 client.4294.objecter handle_osd_map decoding full epoch 18 2023-12-11T20:16:53.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.831+0000 7f87eeffd640 20 client.4294.objecter dump_active .. 0 homeless 2023-12-11T20:16:53.840 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.841+0000 7f87eeffd640 1 -- 172.21.15.86:0/4144868844 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87f8055740 con 0x7f88001429f0 2023-12-11T20:16:53.840 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.841+0000 7f87eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:53.840 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:53.841+0000 7f87eeffd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:54.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:54.020+0000 7f880494e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-11T20:16:54.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:54.021+0000 7f880494e640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-11T20:16:54.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:54.021+0000 7f880494e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:54.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:54.021+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f88000aba80 con 0x7f88001429f0 2023-12-11T20:16:55.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.011+0000 7f87eeffd640 1 -- 172.21.15.86:0/4144868844 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f88000aba80 con 0x7f88001429f0 2023-12-11T20:16:55.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.011+0000 7f87eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-11T20:16:55.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.011+0000 7f87eeffd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-12-11T20:16:55.011 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'rbd' created 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 10 librados: watch_flush enter 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 10 librados: watch_flush exit 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 20 client.4294.objecter shutdown clearing up homeless session... 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 10 client.4294.objecter successfully canceled tick 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f87e006c820 msgr2=0x7f87e006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f87e006c820 0x7f87e006ece0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f87f0036000 tx=0x7f87f003d040 comp rx=0 tx=0).stop 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 10 monclient: shutdown 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:55.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88001429f0 msgr2=0x7f88001d72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88001429f0 0x7f88001d72f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f87f800efc0 tx=0x7f87f800c490 comp rx=0 tx=0).stop 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 shutdown_connections 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f87e006c820 0x7f87e006ece0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88001429f0 0x7f88001d72f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88000b21a0 0x7f88001d6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 --2- 172.21.15.86:0/4144868844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88000a8930 0x7f8800144b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 >> 172.21.15.86:0/4144868844 conn(0x7f88000b82f0 msgr2=0x7f8800073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 shutdown_connections 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 -- 172.21.15.86:0/4144868844 wait complete. 2023-12-11T20:16:55.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.013+0000 7f880494e640 1 librados: shutdown 2023-12-11T20:16:55.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-11T20:16:55.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.128+0000 7ffabb0c9640 10 monclient: get_monmap_and_config 2023-12-11T20:16:55.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.128+0000 7ffabb0c9640 10 monclient: build_initial_monmap 2023-12-11T20:16:55.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.128+0000 7ffabb0c9640 10 monclient: monmap: 2023-12-11T20:16:55.126 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:55.126 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:55.126 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:55.128583+0000 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:55.128583+0000 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:55.127 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:55.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.128+0000 7ffabb0c9640 1 Processor -- start 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.128+0000 7ffabb0c9640 1 -- start start 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.128+0000 7ffabb0c9640 10 monclient: init 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffab40b0a80 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7ffab4072860 0x7ffab40701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient(hunting): picked mon.noname-c con 0x7ffab4072860 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab413bd10 0x7ffab413c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient(hunting): picked mon.noname-b con 0x7ffab413bd10 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab413c700 con 0x7ffab40b0a80 2023-12-11T20:16:55.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab40b0e60 con 0x7ffab4072860 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab413f1d0 con 0x7ffab413bd10 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.129+0000 7ffabb0c9640 10 monclient(hunting): authenticate will time out at 1501.458687s 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.130+0000 7ffab9666640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab413bd10 0x7ffab413c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.130+0000 7ffab9666640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab413bd10 0x7ffab413c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44022/0 (socket says 172.21.15.86:44022) 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.130+0000 7ffab9666640 1 -- 172.21.15.86:0/2176846469 learned_addr learned my addr 172.21.15.86:0/2176846469 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.130+0000 7ffab9666640 10 monclient(hunting): get_auth_request con 0x7ffab413bd10 auth_method 0 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.130+0000 7ffab9666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.130+0000 7ffab9666640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:55.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.130+0000 7ffab9666640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.130+0000 7ffab3fff640 1 --2- 172.21.15.86:0/2176846469 >> v2:172.21.15.133:3300/0 conn(0x7ffab4072860 0x7ffab40701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab3fff640 10 monclient(hunting): get_auth_request con 0x7ffab4072860 auth_method 0 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab3fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab9666640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab9666640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab9666640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.131+0000 7ffab3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2023-12-11T20:16:55.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 1 -- 172.21.15.86:0/2176846469 >> v2:172.21.15.133:3300/0 conn(0x7ffab4072860 msgr2=0x7ffab40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:55.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 1 --2- 172.21.15.86:0/2176846469 >> v2:172.21.15.133:3300/0 conn(0x7ffab4072860 0x7ffab40701c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 1 -- 172.21.15.86:0/2176846469 >> v1:172.21.15.133:6789/0 conn(0x7ffab40b0a80 legacy=0x7ffab413eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:55.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 10 monclient: _finish_hunting 0 2023-12-11T20:16:55.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 1 monclient: found mon.noname-b 2023-12-11T20:16:55.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:55.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:55.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.132+0000 7ffab9666640 1 -- 172.21.15.86:0/2176846469 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab4072160 con 0x7ffab413bd10 2023-12-11T20:16:55.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.134+0000 7ffab9666640 1 --2- 172.21.15.86:0/2176846469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab413bd10 0x7ffab413c1c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009e50 tx=0x7ffaac02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b166785e3739a854 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:55.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 1 -- 172.21.15.86:0/2176846469 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac02cb00 con 0x7ffab413bd10 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: dump: 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: _finish_auth 0 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: _check_auth_tickets 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 1 -- 172.21.15.86:0/2176846469 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaac02cc60 con 0x7ffab413bd10 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 5 monclient: authenticate success, global_id 4229 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:55.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 4 set_mon_vals no callback set 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 1 -- 172.21.15.86:0/2176846469 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0343f0 con 0x7ffab413bd10 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffab37fe640 10 monclient: dump: 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 10 monclient: shutdown 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2176846469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab413bd10 msgr2=0x7ffab413c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2176846469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab413bd10 0x7ffab413c1c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009e50 tx=0x7ffaac02bf60 comp rx=0 tx=0).stop 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2176846469 shutdown_connections 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2176846469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab413bd10 0x7ffab413c1c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2176846469 >> v2:172.21.15.133:3300/0 conn(0x7ffab4072860 0x7ffab40701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2176846469 >> 172.21.15.86:0/2176846469 conn(0x7ffab406ac40 msgr2=0x7ffab40b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2176846469 shutdown_connections 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.138+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2176846469 wait complete. 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 10 monclient: build_initial_monmap 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 10 monclient: monmap: 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:55.142 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:55.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 1 librados: starting msgr at 2023-12-11T20:16:55.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 1 librados: starting objecter 2023-12-11T20:16:55.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 1 Processor -- start 2023-12-11T20:16:55.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 1 -- start start 2023-12-11T20:16:55.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 1 librados: setting wanted keys 2023-12-11T20:16:55.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 1 librados: calling monclient init 2023-12-11T20:16:55.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.139+0000 7ffabb0c9640 10 monclient: init 2023-12-11T20:16:55.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:55.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:16:55.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffab4072860 0x7ffab406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:55.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient(hunting): picked mon.noname-b con 0x7ffab4072860 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:55.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab40b0a80 0x7ffab41da770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:55.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient(hunting): picked mon.noname-c con 0x7ffab40b0a80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:55.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab413bd10 0x7ffab4131790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient(hunting): picked mon.noname-a con 0x7ffab413bd10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab4137cd0 con 0x7ffab413bd10 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab41db160 con 0x7ffab4072860 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab4137f20 con 0x7ffab40b0a80 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:55.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.140+0000 7ffabb0c9640 10 monclient(hunting): authenticate will time out at 1501.469755s 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffab4072860 0x7ffab406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab40b0a80 0x7ffab41da770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab9666640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab413bd10 0x7ffab4131790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffab4072860 0x7ffab406e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:33360/0 (socket says 172.21.15.86:33360) 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 1 -- 172.21.15.86:0/2146175623 learned_addr learned my addr 172.21.15.86:0/2146175623 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 10 monclient(hunting): get_auth_request con 0x7ffab4072860 auth_method 0 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab3fff640 10 monclient(hunting): get_auth_request con 0x7ffab40b0a80 auth_method 0 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab3fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab9666640 10 monclient(hunting): get_auth_request con 0x7ffab413bd10 auth_method 0 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab9666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab9666640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab9666640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab8e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.141+0000 7ffab3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.142+0000 7ffab9666640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.142+0000 7ffab9666640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.142+0000 7ffab9666640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 10 monclient(hunting): handle_auth_done global_id 4303 payload 824 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 1 -- 172.21.15.86:0/2146175623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab40b0a80 msgr2=0x7ffab41da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab40b0a80 0x7ffab41da770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 1 -- 172.21.15.86:0/2146175623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab413bd10 msgr2=0x7ffab4131790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab413bd10 0x7ffab4131790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 10 monclient: _finish_hunting 0 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 1 monclient: found mon.noname-b 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.144+0000 7ffab8e65640 1 -- 172.21.15.86:0/2146175623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac009b30 con 0x7ffab4072860 2023-12-11T20:16:55.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.146+0000 7ffab8e65640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffab4072860 0x7ffab406e2b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800c400 tx=0x7ffaa800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:55.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.146+0000 7ffab17fa640 10 client.?.objecter ms_handle_connect 0x7ffab4072860 2023-12-11T20:16:55.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.146+0000 7ffab17fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:55.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 1 -- 172.21.15.86:0/2146175623 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80052d0 con 0x7ffab4072860 2023-12-11T20:16:55.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: dump: 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:55.154 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: _finish_auth 0 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: _check_auth_tickets 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 1 -- 172.21.15.86:0/2146175623 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa8005450 con 0x7ffab4072860 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 5 monclient: authenticate success, global_id 4303 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 10 monclient: _renew_subs 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab41dacb0 con 0x7ffab4072860 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 1 -- 172.21.15.86:0/2146175623 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8007260 con 0x7ffab4072860 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab17fa640 10 monclient: dump: 2023-12-11T20:16:55.155 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffab2ffd640 4 set_mon_vals no callback set 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 10 monclient: _renew_subs 2023-12-11T20:16:55.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:55.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab4131eb0 con 0x7ffab4072860 2023-12-11T20:16:55.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.150+0000 7ffabb0c9640 1 librados: init done 2023-12-11T20:16:55.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.151+0000 7ffabb0c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:55.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.151+0000 7ffabb0c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:55.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.151+0000 7ffabb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:55.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.151+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab404e840 con 0x7ffab4072860 2023-12-11T20:16:55.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.169+0000 7ffab17fa640 1 -- 172.21.15.86:0/2146175623 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffaa80073c0 con 0x7ffab4072860 2023-12-11T20:16:55.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.170+0000 7ffab17fa640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ffa9806c870 0x7ffa9806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:55.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.170+0000 7ffab3fff640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ffa9806c870 0x7ffa9806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:55.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.170+0000 7ffab3fff640 10 monclient: get_auth_request con 0x7ffa9806c870 auth_method 0 2023-12-11T20:16:55.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.170+0000 7ffab17fa640 10 client.4303.objecter ms_handle_connect 0x7ffa9806c870 2023-12-11T20:16:55.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.170+0000 7ffab3fff640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ffa9806c870 0x7ffa9806ed30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffab40ad970 tx=0x7ffaa0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:55.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.172+0000 7ffab17fa640 1 -- 172.21.15.86:0/2146175623 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5199+0+0 (secure 0 0 0) 0x7ffaa8007670 con 0x7ffab4072860 2023-12-11T20:16:55.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.172+0000 7ffab17fa640 10 client.4303.objecter ms_dispatch 0x7ffab4000b90 osd_map(19..19 src has 1..19) v4 2023-12-11T20:16:55.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.172+0000 7ffab17fa640 3 client.4303.objecter handle_osd_map got epochs [19,19] > 0 2023-12-11T20:16:55.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.172+0000 7ffab17fa640 3 client.4303.objecter handle_osd_map decoding full epoch 19 2023-12-11T20:16:55.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.172+0000 7ffab17fa640 20 client.4303.objecter dump_active .. 0 homeless 2023-12-11T20:16:55.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.411+0000 7ffab17fa640 1 -- 172.21.15.86:0/2146175623 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffaa8054370 con 0x7ffab4072860 2023-12-11T20:16:55.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.411+0000 7ffab17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:55.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.411+0000 7ffab17fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:55.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.580+0000 7ffabb0c9640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-11T20:16:55.579 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.580+0000 7ffabb0c9640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-11T20:16:55.579 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.580+0000 7ffabb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:55.579 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:55.580+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffab4061ac0 con 0x7ffab4072860 2023-12-11T20:16:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.031+0000 7ffab17fa640 1 -- 172.21.15.86:0/2146175623 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7ffaa80200c0 con 0x7ffab4072860 2023-12-11T20:16:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.031+0000 7ffab17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-11T20:16:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.031+0000 7ffab17fa640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-12-11T20:16:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-11T20:16:56.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 10 librados: watch_flush enter 2023-12-11T20:16:56.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 10 librados: watch_flush exit 2023-12-11T20:16:56.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 20 client.4303.objecter shutdown clearing up homeless session... 2023-12-11T20:16:56.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 10 client.4303.objecter successfully canceled tick 2023-12-11T20:16:56.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ffa9806c870 msgr2=0x7ffa9806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ffa9806c870 0x7ffa9806ed30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffab40ad970 tx=0x7ffaa0009190 comp rx=0 tx=0).stop 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 10 monclient: shutdown 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffab4072860 msgr2=0x7ffab406e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.033+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffab4072860 0x7ffab406e2b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800c400 tx=0x7ffaa800c8c0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 shutdown_connections 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ffa9806c870 0x7ffa9806ed30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffab40b0a80 0x7ffab41da770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab413bd10 0x7ffab4131790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 --2- 172.21.15.86:0/2146175623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffab4072860 0x7ffab406e2b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 >> 172.21.15.86:0/2146175623 conn(0x7ffab406ac40 msgr2=0x7ffab4138b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 shutdown_connections 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 -- 172.21.15.86:0/2146175623 wait complete. 2023-12-11T20:16:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.034+0000 7ffabb0c9640 1 librados: shutdown 2023-12-11T20:16:56.044 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-11T20:16:56.044 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-11T20:16:56.045 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-11T20:16:56.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-11T20:16:56.045 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-11T20:16:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.153+0000 7f06a1bd0640 10 monclient: get_monmap_and_config 2023-12-11T20:16:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.153+0000 7f06a1bd0640 10 monclient: build_initial_monmap 2023-12-11T20:16:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.153+0000 7f06a1bd0640 10 monclient: monmap: 2023-12-11T20:16:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:56.153609+0000 2023-12-11T20:16:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:56.153609+0000 2023-12-11T20:16:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:56.152 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:56.152 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:56.152 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:56.152 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:56.152 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:56.152 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:56.152 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:56.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.153+0000 7f06a1bd0640 1 Processor -- start 2023-12-11T20:16:56.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.153+0000 7f06a1bd0640 1 -- start start 2023-12-11T20:16:56.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.153+0000 7f06a1bd0640 10 monclient: init 2023-12-11T20:16:56.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f069c0a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f069c1461c0 0x7f069c0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f069c1461c0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f069c13f0f0 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c145620 con 0x7f069c0a8a20 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0aace0 con 0x7f069c13f0f0 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069c13fc60 con 0x7f069c1461c0 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.154+0000 7f06a1bd0640 10 monclient(hunting): authenticate will time out at 1502.483656s 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.155+0000 7f069b7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f069c0a8a20 0x7f069c145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48610/0 (socket says 172.21.15.86:48610) 2023-12-11T20:16:56.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.155+0000 7f069b7fe640 1 -- 172.21.15.86:0/3647446584 learned_addr learned my addr 172.21.15.86:0/3647446584 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.156+0000 7f069affd640 1 --2- 172.21.15.86:0/3647446584 >> v2:172.21.15.133:3300/0 conn(0x7f069c1461c0 0x7f069c0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069affd640 10 monclient(hunting): get_auth_request con 0x7f069c1461c0 auth_method 0 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069affd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069a7fc640 1 -- 172.21.15.86:0/3647446584 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442636509 0 0) 0x7f069c145620 con 0x7f069c0a8a20 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069a7fc640 10 monclient(hunting): _negotiate 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069a7fc640 10 monclient(hunting): my global_id is 4257 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.157+0000 7f069a7fc640 1 -- 172.21.15.86:0/3647446584 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688003520 con 0x7f069c0a8a20 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.158+0000 7f069affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.158+0000 7f069affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:56.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.158+0000 7f069affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:56.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.160+0000 7f069affd640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2023-12-11T20:16:56.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.160+0000 7f069affd640 1 -- 172.21.15.86:0/3647446584 >> v1:172.21.15.133:6789/0 conn(0x7f069c13f0f0 legacy=0x7f069c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.160+0000 7f069affd640 1 -- 172.21.15.86:0/3647446584 >> v1:172.21.15.86:6789/0 conn(0x7f069c0a8a20 legacy=0x7f069c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.160+0000 7f069affd640 10 monclient: _finish_hunting 0 2023-12-11T20:16:56.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.160+0000 7f069affd640 1 monclient: found mon.noname-c 2023-12-11T20:16:56.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.160+0000 7f069affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:56.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.160+0000 7f069affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:56.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.160+0000 7f069affd640 1 -- 172.21.15.86:0/3647446584 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0aafc0 con 0x7f069c1461c0 2023-12-11T20:16:56.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.164+0000 7f069affd640 1 --2- 172.21.15.86:0/3647446584 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c1461c0 0x7f069c0a8e00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f06840097a0 tx=0x7f068402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d5544029c759ecc server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 1 -- 172.21.15.86:0/3647446584 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684030610 con 0x7f069c1461c0 2023-12-11T20:16:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: dump: 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 1 -- 172.21.15.86:0/3647446584 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0684030770 con 0x7f069c1461c0 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 1 -- 172.21.15.86:0/3647446584 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684030b00 con 0x7f069c1461c0 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f06a1bd0640 5 monclient: authenticate success, global_id 4306 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f06a1bd0640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f06a1bd0640 4 set_mon_vals no callback set 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.168+0000 7f069a7fc640 10 monclient: dump: 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:56.169 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:56.169 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:56.169 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:56.169 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:56.169 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:56.169 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:56.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 10 monclient: shutdown 2023-12-11T20:16:56.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:56.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 1 -- 172.21.15.86:0/3647446584 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c1461c0 msgr2=0x7f069c0a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 1 --2- 172.21.15.86:0/3647446584 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c1461c0 0x7f069c0a8e00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f06840097a0 tx=0x7f068402b960 comp rx=0 tx=0).stop 2023-12-11T20:16:56.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 1 -- 172.21.15.86:0/3647446584 shutdown_connections 2023-12-11T20:16:56.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 1 --2- 172.21.15.86:0/3647446584 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c1461c0 0x7f069c0a8e00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 1 -- 172.21.15.86:0/3647446584 >> 172.21.15.86:0/3647446584 conn(0x7f069c0b7890 msgr2=0x7f069c0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 1 -- 172.21.15.86:0/3647446584 shutdown_connections 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.169+0000 7f06a1bd0640 1 -- 172.21.15.86:0/3647446584 wait complete. 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 10 monclient: build_initial_monmap 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 10 monclient: monmap: 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 1 librados: starting msgr at 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 1 librados: starting objecter 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 1 Processor -- start 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 1 -- start start 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 1 librados: setting wanted keys 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 1 librados: calling monclient init 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.170+0000 7f06a1bd0640 10 monclient: init 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f069c0a8a20 0x7f069c134260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f069c0a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c13f0f0 0x7f069c1422e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient(hunting): picked mon.noname-b con 0x7f069c13f0f0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f069c1461c0 0x7f069c140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f069c1461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c142df0 con 0x7f069c1461c0 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c1419e0 con 0x7f069c13f0f0 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c141be0 con 0x7f069c0a8a20 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.171+0000 7f06a1bd0640 10 monclient(hunting): authenticate will time out at 1502.500691s 2023-12-11T20:16:56.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f069c1461c0 0x7f069c140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f069c0a8a20 0x7f069c134260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069affd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c13f0f0 0x7f069c1422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f069c0a8a20 0x7f069c134260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44032/0 (socket says 172.21.15.86:44032) 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069b7fe640 1 -- 172.21.15.86:0/579975294 learned_addr learned my addr 172.21.15.86:0/579975294 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069b7fe640 10 monclient(hunting): get_auth_request con 0x7f069c0a8a20 auth_method 0 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069affd640 10 monclient(hunting): get_auth_request con 0x7f069c13f0f0 auth_method 0 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069affd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069bfff640 10 monclient(hunting): get_auth_request con 0x7f069c1461c0 auth_method 0 2023-12-11T20:16:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069bfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.172+0000 7f069bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:56.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.173+0000 7f069affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:56.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2023-12-11T20:16:56.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 1 -- 172.21.15.86:0/579975294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f069c0a8a20 msgr2=0x7f069c134260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f069c0a8a20 0x7f069c134260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 1 -- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c13f0f0 msgr2=0x7f069c1422e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c13f0f0 0x7f069c1422e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 1 monclient: found mon.noname-a 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.176+0000 7f069bfff640 1 -- 172.21.15.86:0/579975294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684009400 con 0x7f069c1461c0 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.178+0000 7f069bfff640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f069c1461c0 0x7f069c140090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f069400c690 tx=0x7f069400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.178+0000 7f0693fff640 10 client.?.objecter ms_handle_connect 0x7f069c1461c0 2023-12-11T20:16:56.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.178+0000 7f0693fff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 1 -- 172.21.15.86:0/579975294 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694003b70 con 0x7f069c1461c0 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: dump: 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:56.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: _finish_auth 0 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 1 -- 172.21.15.86:0/579975294 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0694003cd0 con 0x7f069c1461c0 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f06a1bd0640 5 monclient: authenticate success, global_id 4260 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f06a1bd0640 10 monclient: _renew_subs 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f06a1bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c142820 con 0x7f069c1461c0 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 1 -- 172.21.15.86:0/579975294 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694005310 con 0x7f069c1461c0 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0693fff640 10 monclient: dump: 2023-12-11T20:16:56.184 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.183+0000 7f0699ffb640 4 set_mon_vals no callback set 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 10 monclient: _renew_subs 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c1407b0 con 0x7f069c1461c0 2023-12-11T20:16:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 1 librados: init done 2023-12-11T20:16:56.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:56.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:56.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:56.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.184+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c13fdb0 con 0x7f069c1461c0 2023-12-11T20:16:56.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.470+0000 7f0693fff640 1 -- 172.21.15.86:0/579975294 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0694005470 con 0x7f069c1461c0 2023-12-11T20:16:56.469 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.470+0000 7f0693fff640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f067c06c820 0x7f067c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:56.469 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.470+0000 7f069b7fe640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f067c06c820 0x7f067c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:56.469 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.470+0000 7f069b7fe640 10 monclient: get_auth_request con 0x7f067c06c820 auth_method 0 2023-12-11T20:16:56.469 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.471+0000 7f069b7fe640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f067c06c820 0x7f067c06ece0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f068c0011e0 tx=0x7f068c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:56.469 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.471+0000 7f0693fff640 10 client.4260.objecter ms_handle_connect 0x7f067c06c820 2023-12-11T20:16:56.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.483+0000 7f0693fff640 1 -- 172.21.15.86:0/579975294 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f06940589a0 con 0x7f069c1461c0 2023-12-11T20:16:56.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.483+0000 7f0693fff640 10 client.4260.objecter ms_dispatch 0x7f069c000b90 osd_map(20..20 src has 1..20) v4 2023-12-11T20:16:56.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.483+0000 7f0693fff640 3 client.4260.objecter handle_osd_map got epochs [20,20] > 0 2023-12-11T20:16:56.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.483+0000 7f0693fff640 3 client.4260.objecter handle_osd_map decoding full epoch 20 2023-12-11T20:16:56.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.483+0000 7f0693fff640 20 client.4260.objecter dump_active .. 0 homeless 2023-12-11T20:16:56.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.483+0000 7f0693fff640 1 -- 172.21.15.86:0/579975294 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0694054ce0 con 0x7f069c1461c0 2023-12-11T20:16:56.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.483+0000 7f0693fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:56.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.483+0000 7f0693fff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:56.677 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:09.786 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:56.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.704+0000 7f06a1bd0640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-12-11T20:16:56.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.704+0000 7f06a1bd0640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-11T20:16:56.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.704+0000 7f06a1bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:56.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.704+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f069c13fdb0 con 0x7f069c1461c0 2023-12-11T20:16:56.733 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:09.842 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:56.765 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:09.834 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:56.767 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:09.876 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:56.817 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:09.886 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:56.822 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:09.930 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.973+0000 7f0693fff640 1 -- 172.21.15.86:0/579975294 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171038 (secure 0 0 0) 0x7f069c13fdb0 con 0x7f069c1461c0 2023-12-11T20:16:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.973+0000 7f0693fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-11T20:16:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.973+0000 7f0693fff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:56.974 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 10 librados: watch_flush enter 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 10 librados: watch_flush exit 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 20 client.4260.objecter shutdown clearing up homeless session... 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 10 client.4260.objecter successfully canceled tick 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f067c06c820 msgr2=0x7f067c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f067c06c820 0x7f067c06ece0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f068c0011e0 tx=0x7f068c009040 comp rx=0 tx=0).stop 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 10 monclient: shutdown 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f069c1461c0 msgr2=0x7f069c140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f069c1461c0 0x7f069c140090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f069400c690 tx=0x7f069400cb50 comp rx=0 tx=0).stop 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 shutdown_connections 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f067c06c820 0x7f067c06ece0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f069c1461c0 0x7f069c140090 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f069c13f0f0 0x7f069c1422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 --2- 172.21.15.86:0/579975294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f069c0a8a20 0x7f069c134260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 >> 172.21.15.86:0/579975294 conn(0x7f069c0b7890 msgr2=0x7f069c13f550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 shutdown_connections 2023-12-11T20:16:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.976+0000 7f06a1bd0640 1 -- 172.21.15.86:0/579975294 wait complete. 2023-12-11T20:16:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:56.977+0000 7f06a1bd0640 1 librados: shutdown 2023-12-11T20:16:56.986 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":5,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1322765564},{"type":"v1","addr":"172.21.15.133:6801","nonce":1322765564}]},"active_addr":"172.21.15.133:6801/1322765564","active_change":"2023-12-11T20:16:04.273210+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":2940466071}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":3020791504}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":333156421}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":2881577675}]}]} 2023-12-11T20:16:56.989 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-11T20:16:56.990 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-11T20:16:56.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:57.043 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:10.111 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:57.107 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:10.215 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.130+0000 7fa19d50f640 10 monclient: get_monmap_and_config 2023-12-11T20:16:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.130+0000 7fa19d50f640 10 monclient: build_initial_monmap 2023-12-11T20:16:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.130+0000 7fa19d50f640 10 monclient: monmap: 2023-12-11T20:16:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:57.130560+0000 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:57.130560+0000 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:57.129 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:57.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.130+0000 7fa19d50f640 1 Processor -- start 2023-12-11T20:16:57.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.130+0000 7fa19d50f640 1 -- start start 2023-12-11T20:16:57.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.130+0000 7fa19d50f640 10 monclient: init 2023-12-11T20:16:57.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:57.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-12-11T20:16:57.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1980a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:57.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa19813f6e0 0x7fa198142320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:57.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient(hunting): picked mon.noname-b con 0x7fa19813f6e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa1980a8e00 0x7fa198142860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient(hunting): picked mon.noname-a con 0x7fa1980a8e00 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1980b5770 con 0x7fa1980a8a20 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1980aac70 con 0x7fa1980a8e00 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1980aaf50 con 0x7fa19813f6e0 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.131+0000 7fa19d50f640 10 monclient(hunting): authenticate will time out at 1503.460449s 2023-12-11T20:16:57.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.137+0000 7fa196ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fa1980a8a20 0x7fa19813efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:59870/0 (socket says 172.21.15.86:59870) 2023-12-11T20:16:57.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.137+0000 7fa196ffd640 1 -- 172.21.15.86:0/1417496406 learned_addr learned my addr 172.21.15.86:0/1417496406 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:16:57.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.142+0000 7fa1967fc640 1 --2- 172.21.15.86:0/1417496406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa19813f6e0 0x7fa198142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.160+0000 7fa1967fc640 10 monclient(hunting): get_auth_request con 0x7fa19813f6e0 auth_method 0 2023-12-11T20:16:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.160+0000 7fa1967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.160+0000 7fa1967fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.160+0000 7fa1967fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.163+0000 7fa1967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.163+0000 7fa1967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.163+0000 7fa1967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2023-12-11T20:16:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 1 -- 172.21.15.86:0/1417496406 >> v2:172.21.15.86:3300/0 conn(0x7fa1980a8e00 msgr2=0x7fa198142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 1 --2- 172.21.15.86:0/1417496406 >> v2:172.21.15.86:3300/0 conn(0x7fa1980a8e00 0x7fa198142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 1 -- 172.21.15.86:0/1417496406 >> v1:172.21.15.133:6789/0 conn(0x7fa1980a8a20 legacy=0x7fa19813efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 10 monclient: _finish_hunting 0 2023-12-11T20:16:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 1 monclient: found mon.noname-b 2023-12-11T20:16:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.165+0000 7fa1967fc640 1 -- 172.21.15.86:0/1417496406 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1980a9e60 con 0x7fa19813f6e0 2023-12-11T20:16:57.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.172+0000 7fa1967fc640 1 --2- 172.21.15.86:0/1417496406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa19813f6e0 0x7fa198142320 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0097a0 tx=0x7fa18c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d889ec25abb92397 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 1 -- 172.21.15.86:0/1417496406 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c02f050 con 0x7fa19813f6e0 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: dump: 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:57.188 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: _finish_auth 0 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 1 -- 172.21.15.86:0/1417496406 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c031700 con 0x7fa19813f6e0 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 1 -- 172.21.15.86:0/1417496406 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c031a90 con 0x7fa19813f6e0 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa19d50f640 5 monclient: authenticate success, global_id 4238 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa19d50f640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa19d50f640 4 set_mon_vals no callback set 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.189+0000 7fa195ffb640 10 monclient: dump: 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:57.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:57.190 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:57.190 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:57.190 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:57.190 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:57.190 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:57.190 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:57.190 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:57.190 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:57.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 10 monclient: shutdown 2023-12-11T20:16:57.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:57.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 1 -- 172.21.15.86:0/1417496406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa19813f6e0 msgr2=0x7fa198142320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:57.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 1 --2- 172.21.15.86:0/1417496406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa19813f6e0 0x7fa198142320 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0097a0 tx=0x7fa18c02b7a0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 1 -- 172.21.15.86:0/1417496406 shutdown_connections 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 1 --2- 172.21.15.86:0/1417496406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa19813f6e0 0x7fa198142320 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 1 --2- 172.21.15.86:0/1417496406 >> v2:172.21.15.86:3300/0 conn(0x7fa1980a8e00 0x7fa198142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 1 -- 172.21.15.86:0/1417496406 >> 172.21.15.86:0/1417496406 conn(0x7fa1980b7890 msgr2=0x7fa1980b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 1 -- 172.21.15.86:0/1417496406 shutdown_connections 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.190+0000 7fa19d50f640 1 -- 172.21.15.86:0/1417496406 wait complete. 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 10 monclient: build_initial_monmap 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 10 monclient: monmap: 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:57.194 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 1 librados: starting msgr at 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 1 librados: starting objecter 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 1 Processor -- start 2023-12-11T20:16:57.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 1 -- start start 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 1 librados: setting wanted keys 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 1 librados: calling monclient init 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.191+0000 7fa19d50f640 10 monclient: init 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980a8a20 0x7fa1980b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient(hunting): picked mon.noname-a con 0x7fa1980a8a20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980a8e00 0x7fa198144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient(hunting): picked mon.noname-c con 0x7fa1980a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa19813f6e0 0x7fa198136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:57.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient(hunting): picked mon.noname-b con 0x7fa19813f6e0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa198146c40 con 0x7fa1980a8a20 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa198145880 con 0x7fa19813f6e0 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa198145da0 con 0x7fa1980a8e00 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.192+0000 7fa19d50f640 10 monclient(hunting): authenticate will time out at 1503.521694s 2023-12-11T20:16:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.194+0000 7fa1967fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980a8e00 0x7fa198144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.197+0000 7fa1967fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980a8e00 0x7fa198144e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39860/0 (socket says 172.21.15.86:39860) 2023-12-11T20:16:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.197+0000 7fa1967fc640 1 -- 172.21.15.86:0/4268031675 learned_addr learned my addr 172.21.15.86:0/4268031675 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.197+0000 7fa1967fc640 10 monclient(hunting): get_auth_request con 0x7fa1980a8e00 auth_method 0 2023-12-11T20:16:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.197+0000 7fa1967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.197+0000 7fa1967fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.197+0000 7fa1967fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.200+0000 7fa1967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.200+0000 7fa1967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:57.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.200+0000 7fa1967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:57.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 1 -- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa19813f6e0 msgr2=0x7fa198136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa19813f6e0 0x7fa198136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 1 -- 172.21.15.86:0/4268031675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980a8a20 msgr2=0x7fa1980b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980a8a20 0x7fa1980b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 10 monclient: _finish_hunting 0 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 1 monclient: found mon.noname-c 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:57.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.206+0000 7fa1967fc640 1 -- 172.21.15.86:0/4268031675 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c009400 con 0x7fa1980a8e00 2023-12-11T20:16:57.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.213+0000 7fa1967fc640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980a8e00 0x7fa198144e90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa18c000c00 tx=0x7fa18c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:57.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.213+0000 7fa17f7fe640 10 client.?.objecter ms_handle_connect 0x7fa1980a8e00 2023-12-11T20:16:57.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.213+0000 7fa17f7fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:57.213 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:10.322 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 1 -- 172.21.15.86:0/4268031675 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c033a70 con 0x7fa1980a8e00 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: dump: 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: _finish_auth 0 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: _check_auth_tickets 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 1 -- 172.21.15.86:0/4268031675 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c033bf0 con 0x7fa1980a8e00 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 5 monclient: authenticate success, global_id 4241 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 10 monclient: _renew_subs 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:57.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1981453d0 con 0x7fa1980a8e00 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 1 -- 172.21.15.86:0/4268031675 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c03e420 con 0x7fa1980a8e00 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa17f7fe640 10 monclient: dump: 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:57.235 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa1957fa640 4 set_mon_vals no callback set 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 10 monclient: _renew_subs 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa198137190 con 0x7fa1980a8e00 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.234+0000 7fa19d50f640 1 librados: init done 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.235+0000 7fa19d50f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:57.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.235+0000 7fa19d50f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:57.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.235+0000 7fa19d50f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:57.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.235+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1981456a0 con 0x7fa1980a8e00 2023-12-11T20:16:57.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.516+0000 7fa17f7fe640 1 -- 172.21.15.86:0/4268031675 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa18c03e580 con 0x7fa1980a8e00 2023-12-11T20:16:57.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.516+0000 7fa17f7fe640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa18406c820 0x7fa18406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:57.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.517+0000 7fa196ffd640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa18406c820 0x7fa18406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:57.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.517+0000 7fa196ffd640 10 monclient: get_auth_request con 0x7fa18406c820 auth_method 0 2023-12-11T20:16:57.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.517+0000 7fa17f7fe640 10 client.4241.objecter ms_handle_connect 0x7fa18406c820 2023-12-11T20:16:57.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.517+0000 7fa196ffd640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa18406c820 0x7fa18406ece0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa188000f60 tx=0x7fa188007a80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.556+0000 7fa17f7fe640 1 -- 172.21.15.86:0/4268031675 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa18c07ae90 con 0x7fa1980a8e00 2023-12-11T20:16:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.556+0000 7fa17f7fe640 10 client.4241.objecter ms_dispatch 0x7fa198000b90 osd_map(20..20 src has 1..20) v4 2023-12-11T20:16:57.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.556+0000 7fa17f7fe640 3 client.4241.objecter handle_osd_map got epochs [20,20] > 0 2023-12-11T20:16:57.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.556+0000 7fa17f7fe640 3 client.4241.objecter handle_osd_map decoding full epoch 20 2023-12-11T20:16:57.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.556+0000 7fa17f7fe640 20 client.4241.objecter dump_active .. 0 homeless 2023-12-11T20:16:57.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.556+0000 7fa17f7fe640 1 -- 172.21.15.86:0/4268031675 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa18c076ad0 con 0x7fa1980a8e00 2023-12-11T20:16:57.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.556+0000 7fa17f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:57.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.556+0000 7fa17f7fe640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:57.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.787+0000 7fa19d50f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:57.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.787+0000 7fa19d50f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:57.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.787+0000 7fa19d50f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:57.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.787+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa19804e840 con 0x7fa1980a8e00 2023-12-11T20:16:57.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.900+0000 7fa17f7fe640 1 -- 172.21.15.86:0/4268031675 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14675 (secure 0 0 0) 0x7fa18c038070 con 0x7fa1980a8e00 2023-12-11T20:16:57.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.900+0000 7fa17f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:57.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.900+0000 7fa17f7fe640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:57.898 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:16:57.899 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":21,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:56.973973+0000","last_up_change":"2023-12-11T20:16:47.001015+0000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-11T20:16:44.927047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-11T20:16:54.037205+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6821","nonce":1914311689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6826","nonce":1914311689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6831","nonce":1914311689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6829","nonce":1914311689}]},"public_addr":"172.21.15.86:6821/1914311689","cluster_addr":"172.21.15.86:6826/1914311689","heartbeat_back_addr":"172.21.15.86:6831/1914311689","heartbeat_front_addr":"172.21.15.86:6829/1914311689","state":["exists","up"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6809","nonce":4202858331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6811","nonce":4202858331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6815","nonce":4202858331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6813","nonce":4202858331}]},"public_addr":"172.21.15.86:6809/4202858331","cluster_addr":"172.21.15.86:6811/4202858331","heartbeat_back_addr":"172.21.15.86:6815/4202858331","heartbeat_front_addr":"172.21.15.86:6813/4202858331","state":["exists","up"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6817","nonce":406445159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6819","nonce":406445159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6825","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6827","nonce":406445159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6823","nonce":406445159}]},"public_addr":"172.21.15.86:6817/406445159","cluster_addr":"172.21.15.86:6819/406445159","heartbeat_back_addr":"172.21.15.86:6827/406445159","heartbeat_front_addr":"172.21.15.86:6823/406445159","state":["exists","up"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6801","nonce":4064445038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6803","nonce":4064445038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6807","nonce":4064445038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6805","nonce":4064445038}]},"public_addr":"172.21.15.86:6801/4064445038","cluster_addr":"172.21.15.86:6803/4064445038","heartbeat_back_addr":"172.21.15.86:6807/4064445038","heartbeat_front_addr":"172.21.15.86:6805/4064445038","state":["exists","up"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6819","nonce":1433273204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6821","nonce":1433273204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6825","nonce":1433273204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6823","nonce":1433273204}]},"public_addr":"172.21.15.133:6819/1433273204","cluster_addr":"172.21.15.133:6821/1433273204","heartbeat_back_addr":"172.21.15.133:6825/1433273204","heartbeat_front_addr":"172.21.15.133:6823/1433273204","state":["exists","up"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6803","nonce":3582621102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6805","nonce":3582621102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6809","nonce":3582621102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6807","nonce":3582621102}]},"public_addr":"172.21.15.133:6803/3582621102","cluster_addr":"172.21.15.133:6805/3582621102","heartbeat_back_addr":"172.21.15.133:6809/3582621102","heartbeat_front_addr":"172.21.15.133:6807/3582621102","state":["exists","up"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6827","nonce":1611390427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6829","nonce":1611390427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6833","nonce":1611390427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6831","nonce":1611390427}]},"public_addr":"172.21.15.133:6827/1611390427","cluster_addr":"172.21.15.133:6829/1611390427","heartbeat_back_addr":"172.21.15.133:6833/1611390427","heartbeat_front_addr":"172.21.15.133:6831/1611390427","state":["exists","up"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6811","nonce":1136721322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6813","nonce":1136721322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6817","nonce":1136721322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6815","nonce":1136721322}]},"public_addr":"172.21.15.133:6811/1136721322","cluster_addr":"172.21.15.133:6813/1136721322","heartbeat_back_addr":"172.21.15.133:6817/1136721322","heartbeat_front_addr":"172.21.15.133:6815/1136721322","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.469087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.889798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.985199+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.354101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.245493+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:41.942057+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.009608+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:42.326615+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:57.900 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 10 librados: watch_flush enter 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 10 librados: watch_flush exit 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 20 client.4241.objecter shutdown clearing up homeless session... 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 10 client.4241.objecter successfully canceled tick 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa18406c820 msgr2=0x7fa18406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa18406c820 0x7fa18406ece0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa188000f60 tx=0x7fa188007a80 comp rx=0 tx=0).stop 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 10 monclient: shutdown 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980a8e00 msgr2=0x7fa198144e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980a8e00 0x7fa198144e90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa18c000c00 tx=0x7fa18c0046c0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 shutdown_connections 2023-12-11T20:16:57.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa18406c820 0x7fa18406ece0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980a8e00 0x7fa198144e90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa19813f6e0 0x7fa198136a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 --2- 172.21.15.86:0/4268031675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980a8a20 0x7fa1980b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:57.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 >> 172.21.15.86:0/4268031675 conn(0x7fa1980b7890 msgr2=0x7fa1980b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:57.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 shutdown_connections 2023-12-11T20:16:57.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 -- 172.21.15.86:0/4268031675 wait complete. 2023-12-11T20:16:57.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:57.902+0000 7fa19d50f640 1 librados: shutdown 2023-12-11T20:16:57.913 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-11T20:16:57.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-11T20:16:58.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.032+0000 7f6da3a5d640 10 monclient: get_monmap_and_config 2023-12-11T20:16:58.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.032+0000 7f6da3a5d640 10 monclient: build_initial_monmap 2023-12-11T20:16:58.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.032+0000 7f6da3a5d640 10 monclient: monmap: 2023-12-11T20:16:58.032 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:58.032531+0000 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:58.032531+0000 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:58.033 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.032+0000 7f6da3a5d640 1 Processor -- start 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.032+0000 7f6da3a5d640 1 -- start start 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.032+0000 7f6da3a5d640 10 monclient: init 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d9c0a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f6d9c1461c0 0x7f6d9c0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c1461c0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d9c13f0f0 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c145620 con 0x7f6d9c0a8a20 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0aace0 con 0x7f6d9c13f0f0 2023-12-11T20:16:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:58.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d9c13fc60 con 0x7f6d9c1461c0 2023-12-11T20:16:58.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:58.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.033+0000 7f6da3a5d640 10 monclient(hunting): authenticate will time out at 1504.362643s 2023-12-11T20:16:58.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.034+0000 7f6da17f9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6d9c0a8a20 0x7f6d9c145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45242/0 (socket says 172.21.15.86:45242) 2023-12-11T20:16:58.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.034+0000 7f6da17f9640 1 -- 172.21.15.86:0/549955835 learned_addr learned my addr 172.21.15.86:0/549955835 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:16:58.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.035+0000 7f6da0ff8640 1 --2- 172.21.15.86:0/549955835 >> v2:172.21.15.133:3300/0 conn(0x7f6d9c1461c0 0x7f6d9c0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:58.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.035+0000 7f6da0ff8640 10 monclient(hunting): get_auth_request con 0x7f6d9c1461c0 auth_method 0 2023-12-11T20:16:58.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.035+0000 7f6da0ff8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:58.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.035+0000 7f6da0ff8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:58.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.035+0000 7f6da0ff8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057212892 0 0) 0x7f6d9c145620 con 0x7f6d9c0a8a20 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6d93fff640 10 monclient(hunting): _negotiate 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6d93fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6d93fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6d93fff640 10 monclient(hunting): my global_id is 4266 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.036+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d88003520 con 0x7f6d9c0a8a20 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.038+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562141778 0 0) 0x7f6d9c0aace0 con 0x7f6d9c13f0f0 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.038+0000 7f6d93fff640 10 monclient(hunting): _negotiate 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.038+0000 7f6d93fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.038+0000 7f6d93fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.038+0000 7f6d93fff640 10 monclient(hunting): my global_id is 4321 2023-12-11T20:16:58.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.038+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c145620 con 0x7f6d9c13f0f0 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944587754 0 0) 0x7f6d88003520 con 0x7f6d9c0a8a20 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 >> v2:172.21.15.133:3300/0 conn(0x7f6d9c1461c0 msgr2=0x7f6d9c0a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 1 --2- 172.21.15.86:0/549955835 >> v2:172.21.15.133:3300/0 conn(0x7f6d9c1461c0 0x7f6d9c0a8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 >> v1:172.21.15.133:6789/0 conn(0x7f6d9c13f0f0 legacy=0x7f6d9c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 1 monclient: found mon.noname-a-legacy 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2023-12-11T20:16:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0aafc0 con 0x7f6d9c0a8a20 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 10 monclient: _finish_auth 0 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1416536749 0 0) 0x7f6d98001670 con 0x7f6d9c0a8a20 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.040+0000 7f6d93fff640 10 monclient: dump: 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 >> v1:172.21.15.86:6789/0 conn(0x7f6d9c0a8a20 legacy=0x7f6d9c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d88003b50 0x7f6d88003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient(hunting): picked mon.b con 0x7f6d88003b50 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d880047c0 0x7f6d88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient(hunting): picked mon.a con 0x7f6d880047c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient(hunting): picked mon.c con 0x7f6d88005180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8800c050 con 0x7f6d880047c0 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8800c190 con 0x7f6d88003b50 2023-12-11T20:16:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:58.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8800c350 con 0x7f6d88005180 2023-12-11T20:16:58.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:58.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.041+0000 7f6d93fff640 10 monclient(hunting): _finish_auth 0 2023-12-11T20:16:58.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.042+0000 7f6da1ffa640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:58.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.042+0000 7f6da1ffa640 10 monclient(hunting): get_auth_request con 0x7f6d88005180 auth_method 0 2023-12-11T20:16:58.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.042+0000 7f6da1ffa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.043+0000 7f6da1ffa640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.043+0000 7f6da1ffa640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.043+0000 7f6da1ffa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.043+0000 7f6da1ffa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.043+0000 7f6da1ffa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.045+0000 7f6da0ff8640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d880047c0 0x7f6d88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da0ff8640 10 monclient(hunting): get_auth_request con 0x7f6d880047c0 auth_method 0 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da0ff8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da0ff8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da0ff8640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:16:58.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 10 monclient(hunting): handle_auth_done global_id 4266 payload 293 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 1 -- 172.21.15.86:0/549955835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d88003b50 msgr2=0x7f6d88003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d88003b50 0x7f6d88003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 1 -- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d880047c0 msgr2=0x7f6d88004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d880047c0 0x7f6d88004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 10 monclient: _finish_hunting 0 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 1 monclient: found mon.c 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 1 -- 172.21.15.86:0/549955835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d88003520 con 0x7f6d88005180 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 10 monclient: _finish_auth 0 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 10 monclient: _check_auth_tickets 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da1ffa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:58.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da3a5d640 5 monclient: authenticate success, global_id 4266 2023-12-11T20:16:58.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.046+0000 7f6da3a5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:58.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.047+0000 7f6da1ffa640 1 -- 172.21.15.86:0/549955835 reap_dead start 2023-12-11T20:16:58.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.049+0000 7f6da1ffa640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d8800b980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6d94001fd0 tx=0x7f6d880057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6af00b7b3f44f87d server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:58.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9402c770 con 0x7f6d88005180 2023-12-11T20:16:58.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:58.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:58.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 10 monclient: dump: 2023-12-11T20:16:58.266 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:58.266 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:58.266 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.266 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:58.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:58.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:58.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:58.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:58.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:58.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9402c8d0 con 0x7f6d88005180 2023-12-11T20:16:58.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6da3a5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:58.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:58.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6da3a5d640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:58.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 1 -- 172.21.15.86:0/549955835 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9402cc40 con 0x7f6d88005180 2023-12-11T20:16:58.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:58.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6da3a5d640 4 set_mon_vals no callback set 2023-12-11T20:16:58.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:58.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.267+0000 7f6d93fff640 10 monclient: dump: 2023-12-11T20:16:58.269 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:58.269 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:58.269 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.269 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.270 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:58.270 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:58.271 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:58.271 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:58.271 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:58.271 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:58.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 10 monclient: shutdown 2023-12-11T20:16:58.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:58.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 1 -- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 msgr2=0x7f6d8800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:58.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d8800b980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6d94001fd0 tx=0x7f6d880057f0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 1 -- 172.21.15.86:0/549955835 shutdown_connections 2023-12-11T20:16:58.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 1 --2- 172.21.15.86:0/549955835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d8800b980 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 1 -- 172.21.15.86:0/549955835 >> 172.21.15.86:0/549955835 conn(0x7f6d9c0b7890 msgr2=0x7f6d9c0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:58.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 1 -- 172.21.15.86:0/549955835 shutdown_connections 2023-12-11T20:16:58.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 1 -- 172.21.15.86:0/549955835 wait complete. 2023-12-11T20:16:58.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 10 monclient: build_initial_monmap 2023-12-11T20:16:58.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.268+0000 7f6da3a5d640 10 monclient: monmap: 2023-12-11T20:16:58.279 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:58.279 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:58.280 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:58.280 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:58.280 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:58.280 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:58.280 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:58.280 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:58.281 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:58.281 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:58.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.269+0000 7f6da3a5d640 1 librados: starting msgr at 2023-12-11T20:16:58.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.269+0000 7f6da3a5d640 1 librados: starting objecter 2023-12-11T20:16:58.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.269+0000 7f6da3a5d640 1 Processor -- start 2023-12-11T20:16:58.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.269+0000 7f6da3a5d640 1 -- start start 2023-12-11T20:16:58.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.269+0000 7f6da3a5d640 1 librados: setting wanted keys 2023-12-11T20:16:58.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.269+0000 7f6da3a5d640 1 librados: calling monclient init 2023-12-11T20:16:58.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.269+0000 7f6da3a5d640 10 monclient: init 2023-12-11T20:16:58.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:58.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:58.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d9c0ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:58.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d88005180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d9c145100 0x7f6d9c1d7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c145100 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9c1d7840 0x7f6d9c137c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c1d7840 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c1d7cc0 con 0x7f6d9c1d7840 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c1d6e70 con 0x7f6d9c145100 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0acca0 con 0x7f6d88005180 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.270+0000 7f6da3a5d640 10 monclient(hunting): authenticate will time out at 1504.599578s 2023-12-11T20:16:58.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.271+0000 7f6da0ff8640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d9c145100 0x7f6d9c1d7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:58.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.271+0000 7f6da0ff8640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d9c145100 0x7f6d9c1d7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52178/0 (socket says 172.21.15.86:52178) 2023-12-11T20:16:58.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.271+0000 7f6da0ff8640 1 -- 172.21.15.86:0/686634332 learned_addr learned my addr 172.21.15.86:0/686634332 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:58.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.271+0000 7f6da0ff8640 10 monclient(hunting): get_auth_request con 0x7f6d9c145100 auth_method 0 2023-12-11T20:16:58.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.271+0000 7f6da0ff8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:58.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.271+0000 7f6da0ff8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:58.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.271+0000 7f6da0ff8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.272+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.272+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.272+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.275+0000 7f6da17f9640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d9c0ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da17f9640 10 monclient(hunting): get_auth_request con 0x7f6d88005180 auth_method 0 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da17f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da17f9640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da17f9640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 1 -- 172.21.15.86:0/686634332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 msgr2=0x7f6d9c0ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d9c0ab6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 1 -- 172.21.15.86:0/686634332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9c1d7840 msgr2=0x7f6d9c137c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9c1d7840 0x7f6d9c137c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 10 monclient: _finish_hunting 0 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 1 monclient: found mon.noname-b 2023-12-11T20:16:58.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:58.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:58.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.276+0000 7f6da0ff8640 1 -- 172.21.15.86:0/686634332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0acf80 con 0x7f6d9c145100 2023-12-11T20:16:58.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.279+0000 7f6da0ff8640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d9c145100 0x7f6d9c1d7300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c011910 tx=0x7f6d8c011dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:58.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.279+0000 7f6d91ffb640 10 client.?.objecter ms_handle_connect 0x7f6d9c145100 2023-12-11T20:16:58.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.279+0000 7f6d91ffb640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:58.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6d91ffb640 1 -- 172.21.15.86:0/686634332 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c00fa70 con 0x7f6d9c145100 2023-12-11T20:16:58.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6d91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:58.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6d91ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6d91ffb640 10 monclient: dump: 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6d91ffb640 10 monclient: _finish_auth 0 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6d91ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:16:58.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6d91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 5 monclient: authenticate success, global_id 4324 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 10 monclient: _renew_subs 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c0abbf0 con 0x7f6d9c145100 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 10 monclient: _renew_subs 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c1421d0 con 0x7f6d9c145100 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.282+0000 7f6da3a5d640 1 librados: init done 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.283+0000 7f6da3a5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.283+0000 7f6da3a5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.283+0000 7f6da3a5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.283+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c0abe10 con 0x7f6d9c145100 2023-12-11T20:16:58.546 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.547+0000 7f6d91ffb640 1 -- 172.21.15.86:0/686634332 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c00fbd0 con 0x7f6d9c145100 2023-12-11T20:16:58.546 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.548+0000 7f6d91ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:58.546 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.548+0000 7f6d91ffb640 1 -- 172.21.15.86:0/686634332 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c014440 con 0x7f6d9c145100 2023-12-11T20:16:58.546 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.548+0000 7f6d91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:58.546 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.548+0000 7f6d91ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:58.546 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.548+0000 7f6d91ffb640 10 monclient: dump: 2023-12-11T20:16:58.546 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:58.547 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.548+0000 7f6d937fe640 4 set_mon_vals no callback set 2023-12-11T20:16:58.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.549+0000 7f6d91ffb640 1 -- 172.21.15.86:0/686634332 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6d8c01d400 con 0x7f6d9c145100 2023-12-11T20:16:58.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.549+0000 7f6d91ffb640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6d8006c820 0x7f6d8006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:58.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.549+0000 7f6da17f9640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6d8006c820 0x7f6d8006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:58.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.549+0000 7f6da17f9640 10 monclient: get_auth_request con 0x7f6d8006c820 auth_method 0 2023-12-11T20:16:58.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.550+0000 7f6da17f9640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6d8006c820 0x7f6d8006ece0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6d98001860 tx=0x7f6d98001610 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:58.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.550+0000 7f6d91ffb640 10 client.4324.objecter ms_handle_connect 0x7f6d8006c820 2023-12-11T20:16:58.552 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.554+0000 7f6d91ffb640 1 -- 172.21.15.86:0/686634332 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6d8c0595d0 con 0x7f6d9c145100 2023-12-11T20:16:58.552 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.554+0000 7f6d91ffb640 10 client.4324.objecter ms_dispatch 0x7f6d9c000b90 osd_map(21..21 src has 1..21) v4 2023-12-11T20:16:58.552 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.554+0000 7f6d91ffb640 3 client.4324.objecter handle_osd_map got epochs [21,21] > 0 2023-12-11T20:16:58.552 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.554+0000 7f6d91ffb640 3 client.4324.objecter handle_osd_map decoding full epoch 21 2023-12-11T20:16:58.552 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.554+0000 7f6d91ffb640 20 client.4324.objecter dump_active .. 0 homeless 2023-12-11T20:16:58.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.565+0000 7f6d91ffb640 1 -- 172.21.15.86:0/686634332 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d8c055210 con 0x7f6d9c145100 2023-12-11T20:16:58.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.565+0000 7f6d91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:58.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.565+0000 7f6d91ffb640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:58.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.759+0000 7f6da3a5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:58.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.759+0000 7f6da3a5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:58.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.759+0000 7f6da3a5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:58.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.759+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d9c0abde0 con 0x7f6d9c145100 2023-12-11T20:16:58.892 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.894+0000 7f6d91ffb640 1 -- 172.21.15.86:0/686634332 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14662 (secure 0 0 0) 0x7f6d8c058830 con 0x7f6d9c145100 2023-12-11T20:16:58.893 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.894+0000 7f6d91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:16:58.893 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.894+0000 7f6d91ffb640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:16:58.893 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:16:58.893 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":22,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:16:58.715309+0000","last_up_change":"2023-12-11T20:16:47.001015+0000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-11T20:16:44.927047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-11T20:16:54.037205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6821","nonce":1914311689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6826","nonce":1914311689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6831","nonce":1914311689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6829","nonce":1914311689}]},"public_addr":"172.21.15.86:6821/1914311689","cluster_addr":"172.21.15.86:6826/1914311689","heartbeat_back_addr":"172.21.15.86:6831/1914311689","heartbeat_front_addr":"172.21.15.86:6829/1914311689","state":["exists","up"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6809","nonce":4202858331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6811","nonce":4202858331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6815","nonce":4202858331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6813","nonce":4202858331}]},"public_addr":"172.21.15.86:6809/4202858331","cluster_addr":"172.21.15.86:6811/4202858331","heartbeat_back_addr":"172.21.15.86:6815/4202858331","heartbeat_front_addr":"172.21.15.86:6813/4202858331","state":["exists","up"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6817","nonce":406445159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6819","nonce":406445159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6825","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6827","nonce":406445159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6823","nonce":406445159}]},"public_addr":"172.21.15.86:6817/406445159","cluster_addr":"172.21.15.86:6819/406445159","heartbeat_back_addr":"172.21.15.86:6827/406445159","heartbeat_front_addr":"172.21.15.86:6823/406445159","state":["exists","up"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6801","nonce":4064445038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6803","nonce":4064445038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6807","nonce":4064445038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6805","nonce":4064445038}]},"public_addr":"172.21.15.86:6801/4064445038","cluster_addr":"172.21.15.86:6803/4064445038","heartbeat_back_addr":"172.21.15.86:6807/4064445038","heartbeat_front_addr":"172.21.15.86:6805/4064445038","state":["exists","up"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6819","nonce":1433273204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6821","nonce":1433273204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6825","nonce":1433273204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6823","nonce":1433273204}]},"public_addr":"172.21.15.133:6819/1433273204","cluster_addr":"172.21.15.133:6821/1433273204","heartbeat_back_addr":"172.21.15.133:6825/1433273204","heartbeat_front_addr":"172.21.15.133:6823/1433273204","state":["exists","up"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6803","nonce":3582621102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6805","nonce":3582621102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6809","nonce":3582621102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6807","nonce":3582621102}]},"public_addr":"172.21.15.133:6803/3582621102","cluster_addr":"172.21.15.133:6805/3582621102","heartbeat_back_addr":"172.21.15.133:6809/3582621102","heartbeat_front_addr":"172.21.15.133:6807/3582621102","state":["exists","up"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6827","nonce":1611390427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6829","nonce":1611390427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6833","nonce":1611390427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6831","nonce":1611390427}]},"public_addr":"172.21.15.133:6827/1611390427","cluster_addr":"172.21.15.133:6829/1611390427","heartbeat_back_addr":"172.21.15.133:6833/1611390427","heartbeat_front_addr":"172.21.15.133:6831/1611390427","state":["exists","up"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6811","nonce":1136721322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6813","nonce":1136721322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6817","nonce":1136721322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6815","nonce":1136721322}]},"public_addr":"172.21.15.133:6811/1136721322","cluster_addr":"172.21.15.133:6813/1136721322","heartbeat_back_addr":"172.21.15.133:6817/1136721322","heartbeat_front_addr":"172.21.15.133:6815/1136721322","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.469087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.889798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.985199+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.354101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.245493+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:41.942057+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.009608+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:42.326615+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:16:58.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 10 librados: watch_flush enter 2023-12-11T20:16:58.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 10 librados: watch_flush exit 2023-12-11T20:16:58.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 20 client.4324.objecter shutdown clearing up homeless session... 2023-12-11T20:16:58.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 10 client.4324.objecter successfully canceled tick 2023-12-11T20:16:58.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6d8006c820 msgr2=0x7f6d8006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:58.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6d8006c820 0x7f6d8006ece0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6d98001860 tx=0x7f6d98001610 comp rx=0 tx=0).stop 2023-12-11T20:16:58.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 10 monclient: shutdown 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d9c145100 msgr2=0x7f6d9c1d7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d9c145100 0x7f6d9c1d7300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c011910 tx=0x7f6d8c011dd0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 shutdown_connections 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6d8006c820 0x7f6d8006ece0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9c1d7840 0x7f6d9c137c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d9c145100 0x7f6d9c1d7300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 --2- 172.21.15.86:0/686634332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88005180 0x7f6d9c0ab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 >> 172.21.15.86:0/686634332 conn(0x7f6d9c0b7890 msgr2=0x7f6d9c0b6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 shutdown_connections 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.896+0000 7f6da3a5d640 1 -- 172.21.15.86:0/686634332 wait complete. 2023-12-11T20:16:58.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:58.897+0000 7f6da3a5d640 1 librados: shutdown 2023-12-11T20:16:58.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-11T20:16:58.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-11T20:16:58.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-11T20:16:58.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-11T20:16:58.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-11T20:16:58.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-11T20:16:58.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-11T20:16:58.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-11T20:16:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.090+0000 7f979cd90640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.090+0000 7f979cd90640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.090+0000 7f979cd90640 10 monclient: monmap: 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.090751+0000 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.090751+0000 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.092+0000 7f979cd90640 1 Processor -- start 2023-12-11T20:16:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.092+0000 7f979cd90640 1 -- start start 2023-12-11T20:16:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.092+0000 7f979cd90640 10 monclient: init 2023-12-11T20:16:59.100 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-12-11T20:16:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979806f000 0x7f97981436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient(hunting): picked mon.noname-b con 0x7f979806f000 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f979806f3e0 0x7f9798143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient(hunting): picked mon.noname-a con 0x7f979806f3e0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9798070e40 0x7f97980712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient(hunting): picked mon.noname-c con 0x7f9798070e40 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9798070540 con 0x7f979806f3e0 2023-12-11T20:16:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97980adcc0 con 0x7f9798070e40 2023-12-11T20:16:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9798070740 con 0x7f979806f000 2023-12-11T20:16:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.093+0000 7f979cd90640 10 monclient(hunting): authenticate will time out at 1505.422281s 2023-12-11T20:16:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.094+0000 7f9795d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f979806f3e0 0x7f9798143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.094+0000 7f979659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979806f000 0x7f97981436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.094+0000 7f979659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979806f000 0x7f97981436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39888/0 (socket says 172.21.15.86:39888) 2023-12-11T20:16:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.094+0000 7f979659c640 1 -- 172.21.15.86:0/4099252904 learned_addr learned my addr 172.21.15.86:0/4099252904 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.094+0000 7f979659c640 10 monclient(hunting): get_auth_request con 0x7f979806f000 auth_method 0 2023-12-11T20:16:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.094+0000 7f979659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.094+0000 7f979659c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.094+0000 7f979659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9796d9d640 1 --2- 172.21.15.86:0/4099252904 >> v2:172.21.15.133:3300/0 conn(0x7f9798070e40 0x7f97980712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9796d9d640 10 monclient(hunting): get_auth_request con 0x7f9798070e40 auth_method 0 2023-12-11T20:16:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9796d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9796d9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9796d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9795d9b640 10 monclient(hunting): get_auth_request con 0x7f979806f3e0 auth_method 0 2023-12-11T20:16:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9795d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9795d9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.106 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.095+0000 7f9795d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.106 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f979659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.106 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f979659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.106 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f979659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.106 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f9796d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.106 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f9796d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.106 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f9796d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f9795d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f9795d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.096+0000 7f9795d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 10 monclient(hunting): handle_auth_done global_id 4327 payload 274 2023-12-11T20:16:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 1 -- 172.21.15.86:0/4099252904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979806f000 msgr2=0x7f97981436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 1 --2- 172.21.15.86:0/4099252904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979806f000 0x7f97981436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 1 -- 172.21.15.86:0/4099252904 >> v2:172.21.15.86:3300/0 conn(0x7f979806f3e0 msgr2=0x7f9798143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 1 --2- 172.21.15.86:0/4099252904 >> v2:172.21.15.86:3300/0 conn(0x7f979806f3e0 0x7f9798143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.108 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.108 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.108 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.108 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.108 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.098+0000 7f9796d9d640 1 -- 172.21.15.86:0/4099252904 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980ad9e0 con 0x7f9798070e40 2023-12-11T20:16:59.108 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.100+0000 7f9796d9d640 1 --2- 172.21.15.86:0/4099252904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9798070e40 0x7f97980712d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9798071da0 tx=0x7f979002bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103ef9c0ec02471b server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.109 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.099+0000 7fd1a72ec640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.109 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.099+0000 7fd1a72ec640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.109 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.100+0000 7fd1a72ec640 10 monclient: monmap: 2023-12-11T20:16:59.109 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.109 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.109 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.100529+0000 2023-12-11T20:16:59.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.100529+0000 2023-12-11T20:16:59.110 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.110 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.110 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.110 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.110 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.111 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.111 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.111 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.101+0000 7fd1a72ec640 1 Processor -- start 2023-12-11T20:16:59.111 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.101+0000 7fd1a72ec640 1 -- start start 2023-12-11T20:16:59.112 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.101+0000 7fd1a72ec640 10 monclient: init 2023-12-11T20:16:59.112 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7f9b21147640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.112 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7f9b21147640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7f9b21147640 10 monclient: monmap: 2023-12-11T20:16:59.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.113 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.103007+0000 2023-12-11T20:16:59.113 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.103007+0000 2023-12-11T20:16:59.113 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.113 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.114 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.114 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.114 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.114 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.114 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.117 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.117 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-12-11T20:16:59.117 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a00af0a0 0x7fd1a00ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.117 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a00af0a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.117 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1a006ed50 0x7fd1a01522d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.118 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a006ed50 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.118 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fd1a006f1b0 0x7fd1a0152810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.118 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a006f1b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.118 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.118 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a00adcf0 con 0x7fd1a006ed50 2023-12-11T20:16:59.118 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a00af480 con 0x7fd1a006f1b0 2023-12-11T20:16:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a0070750 con 0x7fd1a00af0a0 2023-12-11T20:16:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.102+0000 7fd1a72ec640 10 monclient(hunting): authenticate will time out at 1505.431804s 2023-12-11T20:16:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.103+0000 7fd1a5ae9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1a006ed50 0x7fd1a01522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a5ae9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1a006ed50 0x7fd1a01522d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34788/0 (socket says 172.21.15.86:34788) 2023-12-11T20:16:59.120 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a5ae9640 1 -- 172.21.15.86:0/2630561068 learned_addr learned my addr 172.21.15.86:0/2630561068 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.120 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a5ae9640 10 monclient(hunting): get_auth_request con 0x7fd1a006ed50 auth_method 0 2023-12-11T20:16:59.120 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a5ae9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.120 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a5ae9640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.120 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a5ae9640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.120 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a6aeb640 1 --2- 172.21.15.86:0/2630561068 >> v2:172.21.15.133:3300/0 conn(0x7fd1a006f1b0 0x7fd1a0152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.103+0000 7f9b21147640 1 Processor -- start 2023-12-11T20:16:59.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 1 -- 172.21.15.86:0/4099252904 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979002cb40 con 0x7f9798070e40 2023-12-11T20:16:59.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: dump: 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.122 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 1 -- 172.21.15.86:0/4099252904 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979002cca0 con 0x7f9798070e40 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 1 -- 172.21.15.86:0/4099252904 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9790034660 con 0x7f9798070e40 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979559a640 10 monclient: dump: 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.124 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a6aeb640 10 monclient(hunting): get_auth_request con 0x7fd1a006f1b0 auth_method 0 2023-12-11T20:16:59.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a6aeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a6aeb640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7fd1a6aeb640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.103+0000 7f9b21147640 1 -- start start 2023-12-11T20:16:59.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.103+0000 7f9b21147640 10 monclient: init 2023-12-11T20:16:59.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f9b21147640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f9b21147640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-12-11T20:16:59.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f9b21147640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9b14035070 0x7f9b140c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 10 monclient(hunting): picked mon.noname-c con 0x7f9b14035070 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9b1402b2b0 0x7f9b140c14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1402b2b0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14004a30 0x7f9b140c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 10 monclient(hunting): picked mon.noname-b con 0x7f9b14004a30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1402bc50 con 0x7f9b1402b2b0 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b14035450 con 0x7f9b14035070 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1402b7c0 con 0x7f9b14004a30 2023-12-11T20:16:59.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f9b21147640 10 monclient(hunting): authenticate will time out at 1505.434479s 2023-12-11T20:16:59.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979cd90640 5 monclient: authenticate success, global_id 4327 2023-12-11T20:16:59.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979cd90640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.104+0000 7f979cd90640 4 set_mon_vals no callback set 2023-12-11T20:16:59.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f979cd90640 10 monclient: shutdown 2023-12-11T20:16:59.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f979cd90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f979cd90640 1 -- 172.21.15.86:0/4099252904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9798070e40 msgr2=0x7f97980712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.128 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7f979cd90640 1 --2- 172.21.15.86:0/4099252904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9798070e40 0x7f97980712d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9798071da0 tx=0x7f979002bd90 comp rx=0 tx=0).stop 2023-12-11T20:16:59.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7fd1a5ae9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7fd1a5ae9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7fd1a5ae9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7fd1a6aeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7fd1a6aeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.105+0000 7fd1a6aeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f9b20946640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14004a30 0x7f9b140c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f55ba84b640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f55ba84b640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f55ba84b640 10 monclient: monmap: 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.106815+0000 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.106815+0000 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.131 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.132 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.132 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.132 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f9b1b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9b1402b2b0 0x7f9b140c14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f9b1b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9b1402b2b0 0x7f9b140c14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34796/0 (socket says 172.21.15.86:34796) 2023-12-11T20:16:59.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f9b1b7fe640 1 -- 172.21.15.86:0/2925943795 learned_addr learned my addr 172.21.15.86:0/2925943795 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f9b1b7fe640 10 monclient(hunting): get_auth_request con 0x7f9b1402b2b0 auth_method 0 2023-12-11T20:16:59.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f9b1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f9b1b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f9b1b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7f55ba84b640 1 Processor -- start 2023-12-11T20:16:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7f55ba84b640 1 -- start start 2023-12-11T20:16:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7f55ba84b640 10 monclient: init 2023-12-11T20:16:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f9b20946640 10 monclient(hunting): get_auth_request con 0x7f9b14004a30 auth_method 0 2023-12-11T20:16:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f9b20946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-12-11T20:16:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f55b40718b0 0x7f55b40ae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 10 monclient(hunting): picked mon.noname-c con 0x7f55b40718b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55b40ad720 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55b41526d0 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b4070b60 con 0x7f55b40ad720 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b4070e40 con 0x7f55b41526d0 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b4071c90 con 0x7f55b40718b0 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f55ba84b640 10 monclient(hunting): authenticate will time out at 1505.437586s 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f9b20946640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f9b20946640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 1 -- 172.21.15.86:0/2630561068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a00af0a0 msgr2=0x7fd1a00ad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 1 --2- 172.21.15.86:0/2630561068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a00af0a0 0x7fd1a00ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 1 -- 172.21.15.86:0/2630561068 >> v2:172.21.15.133:3300/0 conn(0x7fd1a006f1b0 msgr2=0x7fd1a0152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 1 --2- 172.21.15.86:0/2630561068 >> v2:172.21.15.133:3300/0 conn(0x7fd1a006f1b0 0x7fd1a0152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7fd1a5ae9640 1 -- 172.21.15.86:0/2630561068 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a0070a30 con 0x7fd1a006ed50 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f979cd90640 1 -- 172.21.15.86:0/4099252904 shutdown_connections 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f979cd90640 1 --2- 172.21.15.86:0/4099252904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9798070e40 0x7f97980712d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f979cd90640 1 --2- 172.21.15.86:0/4099252904 >> v2:172.21.15.86:3300/0 conn(0x7f979806f3e0 0x7f9798143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f979cd90640 1 --2- 172.21.15.86:0/4099252904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979806f000 0x7f97981436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f979cd90640 1 -- 172.21.15.86:0/4099252904 >> 172.21.15.86:0/4099252904 conn(0x7f979806af10 msgr2=0x7f97980b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f979cd90640 1 -- 172.21.15.86:0/4099252904 shutdown_connections 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.106+0000 7f979cd90640 1 -- 172.21.15.86:0/4099252904 wait complete. 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7f979cd90640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7f979cd90640 10 monclient: monmap: 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7f979cd90640 1 librados: starting msgr at 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.107+0000 7f979cd90640 1 librados: starting objecter 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f979cd90640 1 Processor -- start 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f979cd90640 1 -- start start 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f979cd90640 1 librados: setting wanted keys 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f979cd90640 1 librados: calling monclient init 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f979cd90640 10 monclient: init 2023-12-11T20:16:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f979cd90640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f979cd90640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.108+0000 7f979cd90640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f979806f000 0x7f97980ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 10 monclient(hunting): picked mon.noname-b con 0x7f979806f000 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f979806f3e0 0x7f97980ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 10 monclient(hunting): picked mon.noname-a con 0x7f979806f3e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9798070e40 0x7f97980a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 10 monclient(hunting): picked mon.noname-c con 0x7f9798070e40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97980ad720 con 0x7f979806f3e0 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9798149e50 con 0x7f979806f000 2023-12-11T20:16:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97980ab2c0 con 0x7f9798070e40 2023-12-11T20:16:59.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f979cd90640 10 monclient(hunting): authenticate will time out at 1505.438346s 2023-12-11T20:16:59.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.109+0000 7f55b3fff640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f55b40718b0 0x7f55b40ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f55b8de8640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f55b41526d0 0x7f55b41592c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:59894/0 (socket says 172.21.15.86:59894) 2023-12-11T20:16:59.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f9b1bfff640 1 --2- 172.21.15.86:0/2925943795 >> v2:172.21.15.133:3300/0 conn(0x7f9b14035070 0x7f9b140c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f55b8de8640 1 -- 172.21.15.86:0/3330712926 learned_addr learned my addr 172.21.15.86:0/3330712926 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:16:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f9b20946640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f9b20946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f9b20946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f9b1bfff640 10 monclient(hunting): get_auth_request con 0x7f9b14035070 auth_method 0 2023-12-11T20:16:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f9b1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f9b1bfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f9b1bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f979659c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f979806f000 0x7f97980ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f55b3fff640 10 monclient(hunting): get_auth_request con 0x7f55b40718b0 auth_method 0 2023-12-11T20:16:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f55b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f55b3fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.110+0000 7f55b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f9796d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9798070e40 0x7f97980a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f9b1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f9b1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f9b1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f979659c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f979806f000 0x7f97980ad1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52206/0 (socket says 172.21.15.86:52206) 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f979659c640 1 -- 172.21.15.86:0/517550087 learned_addr learned my addr 172.21.15.86:0/517550087 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f979659c640 10 monclient(hunting): get_auth_request con 0x7f979806f000 auth_method 0 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f979659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f979659c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f979659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f9796d9d640 10 monclient(hunting): get_auth_request con 0x7f9798070e40 auth_method 0 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f9796d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f9796d9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.111+0000 7f9796d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.113+0000 7f9b1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.113+0000 7f9b1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.113+0000 7f9b1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.113+0000 7f979659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.113+0000 7f979659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.114+0000 7f979659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.114+0000 7f9b20946640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 1 -- 172.21.15.86:0/2925943795 >> v2:172.21.15.133:3300/0 conn(0x7f9b14035070 msgr2=0x7f9b140c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 1 --2- 172.21.15.86:0/2925943795 >> v2:172.21.15.133:3300/0 conn(0x7f9b14035070 0x7f9b140c0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 1 -- 172.21.15.86:0/2925943795 >> v2:172.21.15.86:3300/0 conn(0x7f9b1402b2b0 msgr2=0x7f9b140c14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 1 --2- 172.21.15.86:0/2925943795 >> v2:172.21.15.86:3300/0 conn(0x7f9b1402b2b0 0x7f9b140c14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.115+0000 7f9b20946640 1 -- 172.21.15.86:0/2925943795 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1402c980 con 0x7f9b14004a30 2023-12-11T20:16:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2023-12-11T20:16:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 1 -- 172.21.15.86:0/517550087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9798070e40 msgr2=0x7f97980a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9798070e40 0x7f97980a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 1 -- 172.21.15.86:0/517550087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f979806f3e0 msgr2=0x7f97980ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f979806f3e0 0x7f97980ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.117+0000 7f979659c640 1 -- 172.21.15.86:0/517550087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9790009a70 con 0x7f979806f000 2023-12-11T20:16:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.118+0000 7fd1a5ae9640 1 --2- 172.21.15.86:0/2630561068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a006ed50 0x7fd1a01522d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd1a008a840 tx=0x7fd19c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1146649d83d60166 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.118+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342253333 0 0) 0x7f55b4070e40 con 0x7f55b41526d0 2023-12-11T20:16:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.118+0000 7f55b2ffd640 10 monclient(hunting): _negotiate 2023-12-11T20:16:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.118+0000 7f55b2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.118+0000 7f55b2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.118+0000 7f55b2ffd640 10 monclient(hunting): my global_id is 4339 2023-12-11T20:16:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.118+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c003520 con 0x7f55b41526d0 2023-12-11T20:16:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.119+0000 7f9b20946640 1 --2- 172.21.15.86:0/2925943795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14004a30 0x7f9b140c3c70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400bcc0 tx=0x7f9b1002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a457e96480ae88 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.122+0000 7f979659c640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f979806f000 0x7f97980ad1e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f978800e870 tx=0x7f978800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.122+0000 7f978effd640 10 client.?.objecter ms_handle_connect 0x7f979806f000 2023-12-11T20:16:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.122+0000 7f978effd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b1affd640 1 -- 172.21.15.86:0/2925943795 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b100316f0 con 0x7f9b14004a30 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b1affd640 10 monclient: dump: 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b1affd640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b1affd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b21147640 5 monclient: authenticate success, global_id 4253 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.123+0000 7f9b21147640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f9b1affd640 1 -- 172.21.15.86:0/2925943795 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b10031850 con 0x7f9b14004a30 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f9b1affd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f9b1affd640 1 -- 172.21.15.86:0/2925943795 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b10031be0 con 0x7f9b14004a30 2023-12-11T20:16:59.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f9b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f9b1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f9b1affd640 10 monclient: dump: 2023-12-11T20:16:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.154 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1719772118 0 0) 0x7f559c003520 con 0x7f55b41526d0 2023-12-11T20:16:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 >> v2:172.21.15.133:3300/0 conn(0x7f55b40718b0 msgr2=0x7f55b40ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 --2- 172.21.15.86:0/3330712926 >> v2:172.21.15.133:3300/0 conn(0x7f55b40718b0 0x7f55b40ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 >> v1:172.21.15.86:6789/0 conn(0x7f55b40ad720 legacy=0x7f55b4151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 monclient: found mon.noname-c-legacy 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.133:6789/0 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4071120 con 0x7f55b41526d0 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55ba84b640 5 monclient: authenticate success, global_id 4339 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55ba84b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1416536749 0 0) 0x7f55a8001670 con 0x7f55b41526d0 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: dump: 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] but i'm connected to v1:172.21.15.133:6789/0, reconnecting 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 >> v1:172.21.15.133:6789/0 conn(0x7f55b41526d0 legacy=0x7f55b41592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 0x7f559c003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient(hunting): picked mon.a con 0x7f559c003760 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f559c0040a0 0x7f559c00b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient(hunting): picked mon.b con 0x7f559c0040a0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f559c00b5a0 0x7f559c00da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient(hunting): picked mon.c con 0x7f559c00b5a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b4071c90 con 0x7f559c003760 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c00e300 con 0x7f559c0040a0 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c00e4c0 con 0x7f559c00b5a0 2023-12-11T20:16:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55b2ffd640 10 monclient(hunting): _finish_auth 0 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f55ba84b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 1 -- 172.21.15.86:0/2630561068 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c030610 con 0x7fd1a006ed50 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: dump: 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.158 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 1 -- 172.21.15.86:0/2630561068 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c030770 con 0x7fd1a006ed50 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 1 -- 172.21.15.86:0/2630561068 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c030b40 con 0x7fd1a006ed50 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a52e8640 10 monclient: dump: 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.160 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f9b21147640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.126+0000 7f9b21147640 4 set_mon_vals no callback set 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 10 monclient: shutdown 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 1 -- 172.21.15.86:0/2925943795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14004a30 msgr2=0x7f9b140c3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 1 --2- 172.21.15.86:0/2925943795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14004a30 0x7f9b140c3c70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400bcc0 tx=0x7f9b1002b790 comp rx=0 tx=0).stop 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 1 -- 172.21.15.86:0/2925943795 shutdown_connections 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 1 --2- 172.21.15.86:0/2925943795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14004a30 0x7f9b140c3c70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 1 --2- 172.21.15.86:0/2925943795 >> v2:172.21.15.86:3300/0 conn(0x7f9b1402b2b0 0x7f9b140c14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 1 --2- 172.21.15.86:0/2925943795 >> v2:172.21.15.133:3300/0 conn(0x7f9b14035070 0x7f9b140c0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7f9b21147640 1 -- 172.21.15.86:0/2925943795 >> 172.21.15.86:0/2925943795 conn(0x7f9b14006510 msgr2=0x7f9b14006920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a72ec640 5 monclient: authenticate success, global_id 4272 2023-12-11T20:16:59.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a72ec640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.127+0000 7fd1a72ec640 4 set_mon_vals no callback set 2023-12-11T20:16:59.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f55b3fff640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 0x7f559c003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f55b3fff640 10 monclient(hunting): get_auth_request con 0x7f559c003760 auth_method 0 2023-12-11T20:16:59.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f55b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f55b3fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f55b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:16:59.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f9b21147640 1 -- 172.21.15.86:0/2925943795 shutdown_connections 2023-12-11T20:16:59.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7fd1a72ec640 10 monclient: shutdown 2023-12-11T20:16:59.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7fd1a72ec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2630561068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a006ed50 msgr2=0x7fd1a01522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7fd1a72ec640 1 --2- 172.21.15.86:0/2630561068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a006ed50 0x7fd1a01522d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd1a008a840 tx=0x7fd19c02b960 comp rx=0 tx=0).stop 2023-12-11T20:16:59.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f55b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f55b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f55b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:16:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7f55b8de8640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f559c00b5a0 0x7f559c00da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2630561068 shutdown_connections 2023-12-11T20:16:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7fd1a72ec640 1 --2- 172.21.15.86:0/2630561068 >> v2:172.21.15.133:3300/0 conn(0x7fd1a006f1b0 0x7fd1a0152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7fd1a72ec640 1 --2- 172.21.15.86:0/2630561068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a006ed50 0x7fd1a01522d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7fd1a72ec640 1 --2- 172.21.15.86:0/2630561068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a00af0a0 0x7fd1a00ad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2630561068 >> 172.21.15.86:0/2630561068 conn(0x7fd1a006a500 msgr2=0x7fd1a006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.128+0000 7f9b21147640 1 -- 172.21.15.86:0/2925943795 wait complete. 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7f9b21147640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7f9b21147640 10 monclient: monmap: 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.167 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7f9b21147640 1 librados: starting msgr at 2023-12-11T20:16:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7f9b21147640 1 librados: starting objecter 2023-12-11T20:16:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7f9b21147640 1 Processor -- start 2023-12-11T20:16:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.129+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2630561068 shutdown_connections 2023-12-11T20:16:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f55b8de8640 10 monclient(hunting): get_auth_request con 0x7f559c00b5a0 auth_method 0 2023-12-11T20:16:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f55b8de8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f55b8de8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f55b8de8640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:16:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2630561068 wait complete. 2023-12-11T20:16:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f66d466b640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f66d466b640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f66d466b640 10 monclient: monmap: 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.131052+0000 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.131052+0000 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.169 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f9b21147640 1 -- start start 2023-12-11T20:16:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f9b21147640 1 librados: setting wanted keys 2023-12-11T20:16:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f9b21147640 1 librados: calling monclient init 2023-12-11T20:16:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f9b21147640 10 monclient: init 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f9b21147640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7f9b21147640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.130+0000 7fd1a72ec640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7fd1a72ec640 10 monclient: monmap: 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.172 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f9b21147640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b14004a30 0x7f9b140bcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7fd1a72ec640 1 librados: starting msgr at 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7fd1a72ec640 1 librados: starting objecter 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f9b21147640 10 monclient(hunting): picked mon.noname-b con 0x7f9b14004a30 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f9b21147640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b1402b2b0 0x7f9b1415cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1402b2b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14035070 0x7f9b140b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 10 monclient(hunting): picked mon.noname-c con 0x7f9b14035070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b140b6aa0 con 0x7f9b1402b2b0 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b140b6d80 con 0x7f9b14004a30 2023-12-11T20:16:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b140b72a0 con 0x7f9b14035070 2023-12-11T20:16:59.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f9b21147640 10 monclient(hunting): authenticate will time out at 1505.461424s 2023-12-11T20:16:59.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978800c940 con 0x7f979806f000 2023-12-11T20:16:59.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: dump: 2023-12-11T20:16:59.175 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.175 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.175 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.175 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.175 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 5 monclient: authenticate success, global_id 4336 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 10 monclient: _renew_subs 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97980abd80 con 0x7f979806f000 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 10 monclient: _renew_subs 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97980aa6f0 con 0x7f979806f000 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f979cd90640 1 librados: init done 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978800caa0 con 0x7f979806f000 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978800ce50 con 0x7f979806f000 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f978effd640 10 monclient: dump: 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.177 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.178 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.178 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.178 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.178 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.178 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.178 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 10 monclient(hunting): handle_auth_done global_id 4339 payload 293 2023-12-11T20:16:59.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 1 -- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f559c00b5a0 msgr2=0x7f559c00da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f559c00b5a0 0x7f559c00da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 1 -- 172.21.15.86:0/3330712926 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f559c0040a0 msgr2=0x7f559c00b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f559c0040a0 0x7f559c00b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 1 monclient: found mon.a 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 1 -- 172.21.15.86:0/3330712926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c003520 con 0x7f559c003760 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7f55b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7fd1a72ec640 1 Processor -- start 2023-12-11T20:16:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7fd1a72ec640 1 -- start start 2023-12-11T20:16:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f55b8de8640 1 -- 172.21.15.86:0/3330712926 reap_dead start 2023-12-11T20:16:59.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7fd1a72ec640 1 librados: setting wanted keys 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7fd1a72ec640 1 librados: calling monclient init 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.131+0000 7fd1a72ec640 10 monclient: init 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1a006ed50 0x7fd1a00ab790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a006ed50 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a006f1b0 0x7fd1a00aafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a006f1b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a00af0a0 0x7fd1a0142a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a00af0a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7f978ffff640 4 set_mon_vals no callback set 2023-12-11T20:16:59.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a00abcd0 con 0x7fd1a00af0a0 2023-12-11T20:16:59.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a013de80 con 0x7fd1a006ed50 2023-12-11T20:16:59.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a00ab510 con 0x7fd1a006f1b0 2023-12-11T20:16:59.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f9b1b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b1402b2b0 0x7f9b1415cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.132+0000 7fd1a72ec640 10 monclient(hunting): authenticate will time out at 1505.461754s 2023-12-11T20:16:59.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f66d466b640 1 Processor -- start 2023-12-11T20:16:59.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7fd1a5ae9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a006f1b0 0x7fd1a00aafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f9b1bfff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b14004a30 0x7f9b140bcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f9b1b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b1402b2b0 0x7f9b1415cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34822/0 (socket says 172.21.15.86:34822) 2023-12-11T20:16:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f9b1b7fe640 1 -- 172.21.15.86:0/620246898 learned_addr learned my addr 172.21.15.86:0/620246898 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f9b1b7fe640 10 monclient(hunting): get_auth_request con 0x7f9b1402b2b0 auth_method 0 2023-12-11T20:16:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f9b1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f9b1b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f9b1b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f9b1bfff640 10 monclient(hunting): get_auth_request con 0x7f9b14004a30 auth_method 0 2023-12-11T20:16:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f9b1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f9b1bfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f9b1bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f979cd90640 10 client.4336.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f979cd90640 15 client.4336.objecter _session_command_op_assign -1 1 2023-12-11T20:16:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f979cd90640 20 client.4336.objecter _get_session osd=-1 returning homeless 2023-12-11T20:16:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f979cd90640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f979cd90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9774002910 con 0x7f979806f000 2023-12-11T20:16:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f66d466b640 1 -- start start 2023-12-11T20:16:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7f66d466b640 10 monclient: init 2023-12-11T20:16:59.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f9b20946640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14035070 0x7f9b140b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7fab96973640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f66d466b640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f66d466b640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7fab96973640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.133+0000 7fab96973640 10 monclient: monmap: 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.134287+0000 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.134287+0000 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.191 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f66d466b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66cc0a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fd1a5ae9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a006f1b0 0x7fd1a00aafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39958/0 (socket says 172.21.15.86:39958) 2023-12-11T20:16:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fd1a5ae9640 1 -- 172.21.15.86:0/2256697928 learned_addr learned my addr 172.21.15.86:0/2256697928 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fd1a5ae9640 10 monclient(hunting): get_auth_request con 0x7fd1a006f1b0 auth_method 0 2023-12-11T20:16:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fd1a5ae9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fd1a5ae9640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fd1a5ae9640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7f66d466b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc1461c0 0x7f66cc0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fab96973640 1 Processor -- start 2023-12-11T20:16:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f66d466b640 10 monclient(hunting): picked mon.noname-b con 0x7f66cc1461c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f66d466b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66cc13f0f0 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f66d466b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66cc145620 con 0x7f66cc0a8a20 2023-12-11T20:16:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f66d466b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66cc0aace0 con 0x7f66cc13f0f0 2023-12-11T20:16:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f66d466b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f66d466b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0aafc0 con 0x7f66cc1461c0 2023-12-11T20:16:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f66d466b640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f66d466b640 10 monclient(hunting): authenticate will time out at 1505.464350s 2023-12-11T20:16:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f9b1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f9b1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f9b1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f9b20946640 10 monclient(hunting): get_auth_request con 0x7f9b14035070 auth_method 0 2023-12-11T20:16:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f9b20946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f9b20946640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f9b20946640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fab96973640 1 -- start start 2023-12-11T20:16:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.134+0000 7fab96973640 10 monclient: init 2023-12-11T20:16:59.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f0d0f175640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f0d0f175640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-12-11T20:16:59.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fab900aeae0 0x7fab9013b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7f0d0f175640 10 monclient: monmap: 2023-12-11T20:16:59.197 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.197 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.197 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.136275+0000 2023-12-11T20:16:59.197 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.136275+0000 2023-12-11T20:16:59.197 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.197 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.198 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.198 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.198 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.198 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.198 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.198 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient(hunting): picked mon.noname-c con 0x7fab900aeae0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab900aeec0 0x7fab90071f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient(hunting): picked mon.noname-b con 0x7fab900aeec0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab9006f800 0x7fab900724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient(hunting): picked mon.noname-a con 0x7fab9006f800 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9013df50 con 0x7fab9006f800 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9013eeb0 con 0x7fab900aeae0 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab90071550 con 0x7fab900aeec0 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.135+0000 7fab96973640 10 monclient(hunting): authenticate will time out at 1505.465073s 2023-12-11T20:16:59.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7f0d0f175640 1 Processor -- start 2023-12-11T20:16:59.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7f0d0f175640 1 -- start start 2023-12-11T20:16:59.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7f0d0f175640 10 monclient: init 2023-12-11T20:16:59.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7f9b20946640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7f9b20946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7f9b20946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7fd1a5ae9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7fd1a5ae9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.136+0000 7fd1a5ae9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0d080a8730 0x7f0d080a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient(hunting): picked mon.noname-a con 0x7f0d080a8730 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f0d081423a0 0x7f0d08142830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient(hunting): picked mon.noname-c con 0x7f0d081423a0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d0813ef70 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d080a9050 con 0x7f0d0813ef70 2023-12-11T20:16:59.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d080a9e10 con 0x7f0d080a8730 2023-12-11T20:16:59.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d080aab40 con 0x7f0d081423a0 2023-12-11T20:16:59.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.137+0000 7f0d0f175640 10 monclient(hunting): authenticate will time out at 1505.466592s 2023-12-11T20:16:59.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2023-12-11T20:16:59.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 1 -- 172.21.15.86:0/2256697928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1a006ed50 msgr2=0x7fd1a00ab790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1a006ed50 0x7fd1a00ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 1 -- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a00af0a0 msgr2=0x7fd1a0142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1a00af0a0 0x7fd1a0142a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.139+0000 7fd1a5ae9640 1 -- 172.21.15.86:0/2256697928 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c0094d0 con 0x7fd1a006f1b0 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.141+0000 7f9b1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.141+0000 7f9b1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.141+0000 7f9b1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.142+0000 7f4ab6e83640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.142+0000 7f4ab6e83640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.142+0000 7f4ab6e83640 10 monclient: monmap: 2023-12-11T20:16:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.143466+0000 2023-12-11T20:16:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.143466+0000 2023-12-11T20:16:59.205 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.206 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.206 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2023-12-11T20:16:59.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 1 -- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14035070 msgr2=0x7f9b140b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b14035070 0x7f9b140b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 1 -- 172.21.15.86:0/620246898 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b14004a30 msgr2=0x7f9b140bcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b14004a30 0x7f9b140bcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f9b1b7fe640 1 -- 172.21.15.86:0/620246898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b10009400 con 0x7f9b1402b2b0 2023-12-11T20:16:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f4ab6e83640 1 Processor -- start 2023-12-11T20:16:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f4ab6e83640 1 -- start start 2023-12-11T20:16:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.143+0000 7f4ab6e83640 10 monclient: init 2023-12-11T20:16:59.209 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.144+0000 7f55b3fff640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 0x7f559c003b60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009730 tx=0x7f559c0047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504c6b97d1273ae7 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.144+0000 7f4ab6e83640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.144+0000 7f4ab6e83640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-12-11T20:16:59.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.144+0000 7f4ab6e83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4aa8038db0 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:59.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.144+0000 7f4ab6e83640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4aa8039190 0x7f4aa80c8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.144+0000 7f4ab6e83640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa8039190 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.144+0000 7f4ab6e83640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80364c0 0x7f4aa80c8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.145+0000 7f4ab6e83640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa80364c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.145+0000 7f4ab6e83640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa80375b0 con 0x7f4aa8038db0 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.145+0000 7f4ab6e83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.145+0000 7f4ab6e83640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa80c7ea0 con 0x7f4aa8039190 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.145+0000 7f4ab6e83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.145+0000 7f4ab6e83640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8035470 con 0x7f4aa80364c0 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.145+0000 7f4ab6e83640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.145+0000 7f4ab6e83640 10 monclient(hunting): authenticate will time out at 1505.474238s 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.147+0000 7f9b1b7fe640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b1402b2b0 0x7f9b1415cc00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c073030 tx=0x7f9b1c067960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.147+0000 7f9b18ff9640 10 client.?.objecter ms_handle_connect 0x7f9b1402b2b0 2023-12-11T20:16:59.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.147+0000 7f9b18ff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.147+0000 7fd1a5ae9640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a006f1b0 0x7fd1a00aafd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.147+0000 7fd196ffd640 10 client.?.objecter ms_handle_connect 0x7fd1a006f1b0 2023-12-11T20:16:59.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.148+0000 7fd196ffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.149+0000 7f0d0d972640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f0d081423a0 0x7f0d08142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.151+0000 7f66d1c06640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc1461c0 0x7f66cc0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc1461c0 0x7f66cc0a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39960/0 (socket says 172.21.15.86:39960) 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 1 -- 172.21.15.86:0/1872150872 learned_addr learned my addr 172.21.15.86:0/1872150872 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 10 monclient(hunting): get_auth_request con 0x7f66cc1461c0 auth_method 0 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.152+0000 7f66d1c06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7f66d1c06640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2023-12-11T20:16:59.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7f66d1c06640 1 -- 172.21.15.86:0/1872150872 >> v1:172.21.15.133:6789/0 conn(0x7f66cc13f0f0 legacy=0x7f66cc13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7fab8f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab900aeec0 0x7fab90071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7f66d1c06640 1 -- 172.21.15.86:0/1872150872 >> v1:172.21.15.86:6789/0 conn(0x7f66cc0a8a20 legacy=0x7f66cc145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7f66d1c06640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7f66d1c06640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7f66d1c06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7f66d1c06640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.154+0000 7f66d1c06640 1 -- 172.21.15.86:0/1872150872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc0b2210 con 0x7f66cc1461c0 2023-12-11T20:16:59.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.155+0000 7fab8f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab900aeec0 0x7fab90071f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39964/0 (socket says 172.21.15.86:39964) 2023-12-11T20:16:59.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.155+0000 7fab8f7fe640 1 -- 172.21.15.86:0/3390618560 learned_addr learned my addr 172.21.15.86:0/3390618560 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.155+0000 7fab8f7fe640 10 monclient(hunting): get_auth_request con 0x7fab900aeec0 auth_method 0 2023-12-11T20:16:59.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.155+0000 7fab8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.155+0000 7fab8f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.155+0000 7fab8f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4030cc0 con 0x7f559c003760 2023-12-11T20:16:59.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 10 monclient: dump: 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a4030e40 con 0x7f559c003760 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 4 set_mon_vals no callback set 2023-12-11T20:16:59.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 1 -- 172.21.15.86:0/3330712926 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4038680 con 0x7f559c003760 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55b2ffd640 10 monclient: dump: 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.220 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7fab8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7fab8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7fab8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 10 monclient: shutdown 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 1 -- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 msgr2=0x7f559c003b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 0x7f559c003b60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009730 tx=0x7f559c0047e0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 1 -- 172.21.15.86:0/3330712926 shutdown_connections 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 1 --2- 172.21.15.86:0/3330712926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 0x7f559c003b60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 1 -- 172.21.15.86:0/3330712926 >> 172.21.15.86:0/3330712926 conn(0x7f55b406a6e0 msgr2=0x7f55b406aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.156+0000 7f55ba84b640 1 -- 172.21.15.86:0/3330712926 shutdown_connections 2023-12-11T20:16:59.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.157+0000 7f55ba84b640 1 -- 172.21.15.86:0/3330712926 wait complete. 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.157+0000 7f55ba84b640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.157+0000 7f55ba84b640 10 monclient: monmap: 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.223 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.157+0000 7f66d1c06640 1 --2- 172.21.15.86:0/1872150872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc1461c0 0x7f66cc0a8e00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f66c80098a0 tx=0x7f66c802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f01c2954514df85b server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.157+0000 7f55ba84b640 1 librados: starting msgr at 2023-12-11T20:16:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.157+0000 7f55ba84b640 1 librados: starting objecter 2023-12-11T20:16:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 1 Processor -- start 2023-12-11T20:16:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 1 -- start start 2023-12-11T20:16:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 1 librados: setting wanted keys 2023-12-11T20:16:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 1 librados: calling monclient init 2023-12-11T20:16:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 10 monclient: init 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 0x7f55b414ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 10 monclient(hunting): picked mon.noname-a con 0x7f559c003760 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41e6f40 0x7f55b414d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 10 monclient(hunting): picked mon.noname-c con 0x7f55b41e6f40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f55b414de80 0x7f55b414e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.158+0000 7f55ba84b640 10 monclient(hunting): picked mon.noname-b con 0x7f55b414de80 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f55ba84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f55ba84b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b414d9d0 con 0x7f559c003760 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f55ba84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f55ba84b640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b414f260 con 0x7f55b414de80 2023-12-11T20:16:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f55ba84b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f55ba84b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b414f4d0 con 0x7f55b41e6f40 2023-12-11T20:16:59.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f55ba84b640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f55ba84b640 10 monclient(hunting): authenticate will time out at 1505.488147s 2023-12-11T20:16:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c039960 con 0x7fd1a006f1b0 2023-12-11T20:16:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: dump: 2023-12-11T20:16:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 5 monclient: authenticate success, global_id 4262 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 10 monclient: _renew_subs 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a0142fb0 con 0x7fd1a006f1b0 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 10 monclient: _renew_subs 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a0141f70 con 0x7fd1a006f1b0 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a72ec640 1 librados: init done 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c039ac0 con 0x7fd1a006f1b0 2023-12-11T20:16:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c039e70 con 0x7fd1a006f1b0 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd1a4ae7640 4 set_mon_vals no callback set 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fd196ffd640 10 monclient: dump: 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.232 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.232 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.232 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.232 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.232 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7f4ab6682640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80364c0 0x7f4aa80c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.159+0000 7fab94f10640 1 --2- 172.21.15.86:0/3390618560 >> v2:172.21.15.86:3300/0 conn(0x7fab9006f800 0x7fab900724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fd1a72ec640 10 client.4262.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fd1a72ec640 15 client.4262.objecter _session_command_op_assign -1 1 2023-12-11T20:16:59.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fd1a72ec640 20 client.4262.objecter _get_session osd=-1 returning homeless 2023-12-11T20:16:59.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fd1a72ec640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fd1a72ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd188002910 con 0x7fd1a006f1b0 2023-12-11T20:16:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fab94f10640 10 monclient(hunting): get_auth_request con 0x7fab9006f800 auth_method 0 2023-12-11T20:16:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fab94f10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fab94f10640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.160+0000 7fab94f10640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab94f10640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab94f10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab94f10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7f0d0d972640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f0d081423a0 0x7f0d08142830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52232/0 (socket says 172.21.15.86:52232) 2023-12-11T20:16:59.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7f0d0d972640 1 -- 172.21.15.86:0/194939982 learned_addr learned my addr 172.21.15.86:0/194939982 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7f0d0d972640 10 monclient(hunting): get_auth_request con 0x7f0d081423a0 auth_method 0 2023-12-11T20:16:59.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7f0d0d972640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7f0d0d972640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7f0d0d972640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9788013470 con 0x7f979806f000 2023-12-11T20:16:59.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f978effd640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f978006c870 0x7f978006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2023-12-11T20:16:59.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 1 -- 172.21.15.86:0/3390618560 >> v2:172.21.15.133:3300/0 conn(0x7fab900aeae0 msgr2=0x7fab9013b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 1 --2- 172.21.15.86:0/3390618560 >> v2:172.21.15.133:3300/0 conn(0x7fab900aeae0 0x7fab9013b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 1 -- 172.21.15.86:0/3390618560 >> v2:172.21.15.86:3300/0 conn(0x7fab9006f800 msgr2=0x7fab900724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 1 --2- 172.21.15.86:0/3390618560 >> v2:172.21.15.86:3300/0 conn(0x7fab9006f800 0x7fab900724d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.161+0000 7fab8f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7fab8f7fe640 1 -- 172.21.15.86:0/3390618560 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab90070600 con 0x7fab900aeec0 2023-12-11T20:16:59.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f4ab5680640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4aa8039190 0x7f4aa80c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f4ab6682640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80364c0 0x7f4aa80c8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39970/0 (socket says 172.21.15.86:39970) 2023-12-11T20:16:59.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f4ab6682640 1 -- 172.21.15.86:0/3437017668 learned_addr learned my addr 172.21.15.86:0/3437017668 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f4ab6682640 10 monclient(hunting): get_auth_request con 0x7f4aa80364c0 auth_method 0 2023-12-11T20:16:59.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f4ab6682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f4ab6682640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f4ab6682640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f9795d9b640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f978006c870 0x7f978006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.162+0000 7f9795d9b640 10 monclient: get_auth_request con 0x7f978006c870 auth_method 0 2023-12-11T20:16:59.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.163+0000 7f9795d9b640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f978006c870 0x7f978006ed30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9784006a90 tx=0x7f9784006a20 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.163+0000 7f978effd640 10 client.4336.objecter ms_handle_connect 0x7f978006c870 2023-12-11T20:16:59.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f4ab6682640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f4ab6682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f4ab6682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9788059110 con 0x7f979806f000 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 10 client.4336.objecter ms_dispatch 0x7f9798000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 3 client.4336.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 3 client.4336.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 10 client.4336.objecter checking command 1 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f9780072c50 0x7f9780075070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 20 client.4336.objecter _get_session s=0x7f9780072580 osd=5 3 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 20 client.4336.objecter put_session s=0x7f9780072580 osd=5 3 2023-12-11T20:16:59.241 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 15 client.4336.objecter _session_command_op_remove -1 1 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 20 client.4336.objecter _get_session s=0x7f9780072580 osd=5 3 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 20 client.4336.objecter get_session s=0x7f9780072580 osd=5 3 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 15 client.4336.objecter _session_command_op_assign 5 1 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 20 client.4336.objecter put_session s=0x7f9780072580 osd=5 4 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 10 client.4336.objecter _send_command 1 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 --> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9780075720 con 0x7f9780072c50 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 20 client.4336.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9788054d50 con 0x7f979806f000 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f978effd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f4ab5680640 10 monclient(hunting): get_auth_request con 0x7f4aa8039190 auth_method 0 2023-12-11T20:16:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f4ab5680640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f4ab5680640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.164+0000 7f4ab5680640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.165+0000 7f0d0d972640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.165+0000 7f0d0d972640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.165+0000 7f0d0d972640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.166+0000 7f4ab5680640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.166+0000 7f4ab5680640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.166+0000 7f4ab5680640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2023-12-11T20:16:59.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 1 -- 172.21.15.86:0/194939982 >> v2:172.21.15.86:3300/0 conn(0x7f0d080a8730 msgr2=0x7f0d080a8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 1 --2- 172.21.15.86:0/194939982 >> v2:172.21.15.86:3300/0 conn(0x7f0d080a8730 0x7f0d080a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 1 -- 172.21.15.86:0/194939982 >> v1:172.21.15.86:6789/0 conn(0x7f0d0813ef70 legacy=0x7f0d0813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.167+0000 7f0d0d972640 1 -- 172.21.15.86:0/194939982 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080aadb0 con 0x7f0d081423a0 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 1 -- 172.21.15.86:0/3437017668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80364c0 msgr2=0x7f4aa80c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 1 --2- 172.21.15.86:0/3437017668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80364c0 0x7f4aa80c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 1 -- 172.21.15.86:0/3437017668 >> v1:172.21.15.86:6789/0 conn(0x7f4aa8038db0 legacy=0x7f4aa80c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.168+0000 7f4ab5680640 1 -- 172.21.15.86:0/3437017668 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa80c5970 con 0x7f4aa8039190 2023-12-11T20:16:59.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c08f070 con 0x7f9b1402b2b0 2023-12-11T20:16:59.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 10 monclient: dump: 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c06ccf0 con 0x7f9b1402b2b0 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b18ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.170+0000 7f9b19ffb640 4 set_mon_vals no callback set 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 5 monclient: authenticate success, global_id 4278 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c072370 con 0x7f9b1402b2b0 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b18ff9640 10 monclient: dump: 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.248 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.249 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 10 monclient: _renew_subs 2023-12-11T20:16:59.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b140b7520 con 0x7f9b1402b2b0 2023-12-11T20:16:59.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7fab8f7fe640 1 --2- 172.21.15.86:0/3390618560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab900aeec0 0x7fab90071f90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab900b1270 tx=0x7fab8402fff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b425e115752832c server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 10 monclient: _renew_subs 2023-12-11T20:16:59.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b140ba180 con 0x7f9b1402b2b0 2023-12-11T20:16:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 1 librados: init done 2023-12-11T20:16:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 10 client.4278.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 15 client.4278.objecter _session_command_op_assign -1 1 2023-12-11T20:16:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 20 client.4278.objecter _get_session osd=-1 returning homeless 2023-12-11T20:16:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.171+0000 7f9b21147640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.172+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b140b6c10 con 0x7f9b1402b2b0 2023-12-11T20:16:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.172+0000 7f55b37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41e6f40 0x7f55b414d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.172+0000 7f55b8de8640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f55b414de80 0x7f55b414e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.172+0000 7f0d0d972640 1 --2- 172.21.15.86:0/194939982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d081423a0 0x7f0d08142830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0d08061ea0 tx=0x7f0cf802b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3fa36a4304d10eb server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.172+0000 7f55b8de8640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f55b414de80 0x7f55b414e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52246/0 (socket says 172.21.15.86:52246) 2023-12-11T20:16:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.172+0000 7f55b8de8640 1 -- 172.21.15.86:0/1073111984 learned_addr learned my addr 172.21.15.86:0/1073111984 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.172+0000 7f55b8de8640 10 monclient(hunting): get_auth_request con 0x7f55b414de80 auth_method 0 2023-12-11T20:16:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.173+0000 7f55b8de8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.253 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.173+0000 7f55b8de8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.253 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.173+0000 7f55b8de8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.253 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.173+0000 7f4ab5680640 1 --2- 172.21.15.86:0/3437017668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8039190 0x7f4aa80c8350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097a0 tx=0x7f4aa002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b33a5e94affe48 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.173+0000 7f55b8de8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.173+0000 7f55b8de8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.173+0000 7f55b8de8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.176+0000 7f55b3fff640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 0x7f55b414ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.176+0000 7f55b3fff640 10 monclient(hunting): get_auth_request con 0x7f559c003760 auth_method 0 2023-12-11T20:16:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.176+0000 7f55b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.176+0000 7f55b3fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.176+0000 7f55b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 1 -- 172.21.15.86:0/1073111984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41e6f40 msgr2=0x7f55b414d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41e6f40 0x7f55b414d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 1 -- 172.21.15.86:0/1073111984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 msgr2=0x7f55b414ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f559c003760 0x7f55b414ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.177+0000 7f55b8de8640 1 -- 172.21.15.86:0/1073111984 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b414d620 con 0x7f55b414de80 2023-12-11T20:16:59.256 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.180+0000 7f9796d9d640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f9780072c50 0x7f9780075070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.256 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.180+0000 7f9796d9d640 10 monclient: get_auth_request con 0x7f9780072c50 auth_method 0 2023-12-11T20:16:59.256 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.180+0000 7f55b8de8640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f55b414de80 0x7f55b414e300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f55b4071570 tx=0x7f55a800eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.256 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.180+0000 7f55b0ff9640 10 client.?.objecter ms_handle_connect 0x7f55b414de80 2023-12-11T20:16:59.257 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.180+0000 7f55b0ff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.259 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 1 -- 172.21.15.86:0/1872150872 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8031640 con 0x7f66cc1461c0 2023-12-11T20:16:59.259 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.259 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: dump: 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.260 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 1 -- 172.21.15.86:0/1872150872 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c80317a0 con 0x7f66cc1461c0 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 1 -- 172.21.15.86:0/1872150872 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8031b30 con 0x7f66cc1461c0 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d1405640 10 monclient: dump: 2023-12-11T20:16:59.261 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d466b640 5 monclient: authenticate success, global_id 4265 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d466b640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d466b640 4 set_mon_vals no callback set 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.181+0000 7f66d466b640 10 monclient: shutdown 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.182+0000 7f66d466b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.182+0000 7f66d466b640 1 -- 172.21.15.86:0/1872150872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc1461c0 msgr2=0x7f66cc0a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.182+0000 7f66d466b640 1 --2- 172.21.15.86:0/1872150872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc1461c0 0x7f66cc0a8e00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f66c80098a0 tx=0x7f66c802b8f0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.182+0000 7f66d466b640 1 -- 172.21.15.86:0/1872150872 shutdown_connections 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.182+0000 7f66d466b640 1 --2- 172.21.15.86:0/1872150872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc1461c0 0x7f66cc0a8e00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.182+0000 7f66d466b640 1 -- 172.21.15.86:0/1872150872 >> 172.21.15.86:0/1872150872 conn(0x7f66cc0b7890 msgr2=0x7f66cc0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.182+0000 7f66d466b640 1 -- 172.21.15.86:0/1872150872 shutdown_connections 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.182+0000 7f66d466b640 1 -- 172.21.15.86:0/1872150872 wait complete. 2023-12-11T20:16:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 10 monclient: monmap: 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 1 librados: starting msgr at 2023-12-11T20:16:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 1 librados: starting objecter 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 1 Processor -- start 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 1 -- start start 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 1 librados: setting wanted keys 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 1 librados: calling monclient init 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.183+0000 7f66d466b640 10 monclient: init 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66cc0a8a20 0x7f66cc136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient(hunting): picked mon.noname-b con 0x7f66cc0a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc13f0f0 0x7f66cc1361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient(hunting): picked mon.noname-c con 0x7f66cc13f0f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66cc1461c0 0x7f66cc1411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient(hunting): picked mon.noname-a con 0x7f66cc1461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1408d0 con 0x7f66cc1461c0 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc140ad0 con 0x7f66cc0a8a20 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc136a70 con 0x7f66cc13f0f0 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f66d466b640 10 monclient(hunting): authenticate will time out at 1505.513284s 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f9796d9d640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f9780072c50 0x7f9780075070 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.184+0000 7f978effd640 10 client.4336.objecter ms_handle_connect 0x7f9780072c50 2023-12-11T20:16:59.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 1 -- 172.21.15.86:0/194939982 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8031750 con 0x7f0d081423a0 2023-12-11T20:16:59.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: dump: 2023-12-11T20:16:59.268 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.268 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.268 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.268 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.268 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 1 -- 172.21.15.86:0/194939982 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf80318b0 con 0x7f0d081423a0 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0f175640 5 monclient: authenticate success, global_id 4348 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0f175640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0f175640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 1 -- 172.21.15.86:0/194939982 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8031c40 con 0x7f0d081423a0 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0f175640 4 set_mon_vals no callback set 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.185+0000 7f0d0d171640 10 monclient: dump: 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.270 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 10 monclient: shutdown 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 1 -- 172.21.15.86:0/194939982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d081423a0 msgr2=0x7f0d08142830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 1 --2- 172.21.15.86:0/194939982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d081423a0 0x7f0d08142830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0d08061ea0 tx=0x7f0cf802b7f0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 1 -- 172.21.15.86:0/194939982 shutdown_connections 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 1 --2- 172.21.15.86:0/194939982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d081423a0 0x7f0d08142830 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 1 --2- 172.21.15.86:0/194939982 >> v2:172.21.15.86:3300/0 conn(0x7f0d080a8730 0x7f0d080a8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 1 -- 172.21.15.86:0/194939982 >> 172.21.15.86:0/194939982 conn(0x7f0d08061900 msgr2=0x7f0d080b7700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 1 -- 172.21.15.86:0/194939982 shutdown_connections 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f0d0f175640 1 -- 172.21.15.86:0/194939982 wait complete. 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a800eed0 con 0x7f55b414de80 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: dump: 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a800c610 con 0x7f55b414de80 2023-12-11T20:16:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55ba84b640 5 monclient: authenticate success, global_id 4351 2023-12-11T20:16:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55ba84b640 10 monclient: _renew_subs 2023-12-11T20:16:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55ba84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b40b2160 con 0x7f55b414de80 2023-12-11T20:16:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a800c9a0 con 0x7f55b414de80 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b0ff9640 10 monclient: dump: 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.275 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 10 monclient: _renew_subs 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b40b2620 con 0x7f55b414de80 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.186+0000 7f55b27fc640 4 set_mon_vals no callback set 2023-12-11T20:16:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 1 librados: init done 2023-12-11T20:16:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f66d2407640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66cc0a8a20 0x7f66cc136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f66d2407640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66cc0a8a20 0x7f66cc136f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52258/0 (socket says 172.21.15.86:52258) 2023-12-11T20:16:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f66d2407640 1 -- 172.21.15.86:0/1220750924 learned_addr learned my addr 172.21.15.86:0/1220750924 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f66d2407640 10 monclient(hunting): get_auth_request con 0x7f66cc0a8a20 auth_method 0 2023-12-11T20:16:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f66d2407640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f66d2407640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f66d2407640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f0d0f175640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f0d0f175640 10 monclient: monmap: 2023-12-11T20:16:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 10 client.4351.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 15 client.4351.objecter _session_command_op_assign -1 1 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 20 client.4351.objecter _get_session osd=-1 returning homeless 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.187+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f55b4067d90 con 0x7f55b414de80 2023-12-11T20:16:59.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 librados: starting msgr at 2023-12-11T20:16:59.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 librados: starting objecter 2023-12-11T20:16:59.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 Processor -- start 2023-12-11T20:16:59.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 -- start start 2023-12-11T20:16:59.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 librados: setting wanted keys 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 librados: calling monclient init 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient: init 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d080a8730 0x7f0d08131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient(hunting): picked mon.noname-c con 0x7f0d080a8730 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d0813ef70 0x7f0d081375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0813ef70 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d081423a0 0x7f0d0813fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f66d2407640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f66d2407640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f66d2407640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient(hunting): picked mon.noname-a con 0x7f0d081423a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08137b20 con 0x7f0d081423a0 2023-12-11T20:16:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08141490 con 0x7f0d0813ef70 2023-12-11T20:16:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08132160 con 0x7f0d080a8730 2023-12-11T20:16:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.188+0000 7f0d0f175640 10 monclient(hunting): authenticate will time out at 1505.518113s 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.190+0000 7f66d2c08640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66cc1461c0 0x7f66cc1411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 1 -- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc13f0f0 msgr2=0x7f66cc1361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66cc13f0f0 0x7f66cc1361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 1 -- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66cc1461c0 msgr2=0x7f66cc1411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66cc1461c0 0x7f66cc1411d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.192+0000 7f66d2407640 1 -- 172.21.15.86:0/1220750924 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80094d0 con 0x7f66cc0a8a20 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.199+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd19c04d460 con 0x7fd1a006f1b0 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.199+0000 7fd196ffd640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd18406c820 0x7fd18406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.200+0000 7fd1a62ea640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd18406c820 0x7fd18406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.200+0000 7fd1a62ea640 10 monclient: get_auth_request con 0x7fd18406c820 auth_method 0 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.200+0000 7fd1a62ea640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd18406c820 0x7fd18406ece0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd19800ab90 tx=0x7fd198009250 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.200+0000 7fd196ffd640 10 client.4262.objecter ms_handle_connect 0x7fd18406c820 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd19c03e070 con 0x7fd1a006f1b0 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 10 client.4262.objecter ms_dispatch 0x7fd1a0000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 3 client.4262.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 3 client.4262.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 10 client.4262.objecter checking command 1 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fd184072c00 0x7fd184075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 20 client.4262.objecter _get_session s=0x7fd184072530 osd=2 3 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 20 client.4262.objecter put_session s=0x7fd184072530 osd=2 3 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 15 client.4262.objecter _session_command_op_remove -1 1 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 20 client.4262.objecter _get_session s=0x7fd184072530 osd=2 3 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 20 client.4262.objecter get_session s=0x7fd184072530 osd=2 3 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 15 client.4262.objecter _session_command_op_assign 2 1 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 20 client.4262.objecter put_session s=0x7fd184072530 osd=2 4 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 10 client.4262.objecter _send_command 1 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 --> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1840756d0 con 0x7fd184072c00 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 20 client.4262.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd19c077ea0 con 0x7fd1a006f1b0 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.203+0000 7fd196ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.204+0000 7fd1a6aeb640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fd184072c00 0x7fd184075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.204+0000 7fd1a6aeb640 10 monclient: get_auth_request con 0x7fd184072c00 auth_method 0 2023-12-11T20:16:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 1 -- 172.21.15.86:0/3390618560 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84030da0 con 0x7fab900aeec0 2023-12-11T20:16:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 10 monclient: dump: 2023-12-11T20:16:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.289 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 1 -- 172.21.15.86:0/3390618560 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab84034070 con 0x7fab900aeec0 2023-12-11T20:16:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.205+0000 7fab8effd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.291 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab8effd640 1 -- 172.21.15.86:0/3390618560 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8403a620 con 0x7fab900aeec0 2023-12-11T20:16:59.291 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.291 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.291 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab8effd640 10 monclient: dump: 2023-12-11T20:16:59.291 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.291 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.292 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.292 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.292 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.292 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.292 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.293 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 5 monclient: authenticate success, global_id 4268 2023-12-11T20:16:59.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 4 set_mon_vals no callback set 2023-12-11T20:16:59.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9b1c0724d0 con 0x7f9b1402b2b0 2023-12-11T20:16:59.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 10 monclient: shutdown 2023-12-11T20:16:59.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.295 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 1 -- 172.21.15.86:0/3390618560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab900aeec0 msgr2=0x7fab90071f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.295 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 1 --2- 172.21.15.86:0/3390618560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab900aeec0 0x7fab90071f90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab900b1270 tx=0x7fab8402fff0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.295 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7f9b18ff9640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9b0006c820 0x7f9b0006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 1 -- 172.21.15.86:0/3390618560 shutdown_connections 2023-12-11T20:16:59.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 1 --2- 172.21.15.86:0/3390618560 >> v2:172.21.15.86:3300/0 conn(0x7fab9006f800 0x7fab900724d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 1 --2- 172.21.15.86:0/3390618560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab900aeec0 0x7fab90071f90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 1 --2- 172.21.15.86:0/3390618560 >> v2:172.21.15.133:3300/0 conn(0x7fab900aeae0 0x7fab9013b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 1 -- 172.21.15.86:0/3390618560 >> 172.21.15.86:0/3390618560 conn(0x7fab900b3b20 msgr2=0x7fab900b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.206+0000 7fab96973640 1 -- 172.21.15.86:0/3390618560 shutdown_connections 2023-12-11T20:16:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.207+0000 7fab96973640 1 -- 172.21.15.86:0/3390618560 wait complete. 2023-12-11T20:16:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.207+0000 7f9b1bfff640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9b0006c820 0x7f9b0006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.207+0000 7f9b1bfff640 10 monclient: get_auth_request con 0x7f9b0006c820 auth_method 0 2023-12-11T20:16:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.207+0000 7fab96973640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.207+0000 7fab96973640 10 monclient: monmap: 2023-12-11T20:16:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7fab96973640 1 librados: starting msgr at 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7fab96973640 1 librados: starting objecter 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7f9b18ff9640 10 client.4278.objecter ms_handle_connect 0x7f9b0006c820 2023-12-11T20:16:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7f9b1bfff640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9b0006c820 0x7f9b0006ece0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9b08009a00 tx=0x7f9b08009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7fab96973640 1 Processor -- start 2023-12-11T20:16:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7fab96973640 1 -- start start 2023-12-11T20:16:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7fab96973640 1 librados: setting wanted keys 2023-12-11T20:16:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7fab96973640 1 librados: calling monclient init 2023-12-11T20:16:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.208+0000 7fab96973640 10 monclient: init 2023-12-11T20:16:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:16:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab9006f800 0x7fab901debd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient(hunting): picked mon.noname-c con 0x7fab9006f800 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab900aeae0 0x7fab900a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient(hunting): picked mon.noname-a con 0x7fab900aeae0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab900a93b0 0x7fab90135c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient(hunting): picked mon.noname-b con 0x7fab900a93b0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab901df4a0 con 0x7fab900aeae0 2023-12-11T20:16:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab900aa520 con 0x7fab900a93b0 2023-12-11T20:16:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab900ab260 con 0x7fab9006f800 2023-12-11T20:16:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fab96973640 10 monclient(hunting): authenticate will time out at 1505.538660s 2023-12-11T20:16:59.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7f0d0e173640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d080a8730 0x7f0d08131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7f0d0e173640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d080a8730 0x7f0d08131660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39998/0 (socket says 172.21.15.86:39998) 2023-12-11T20:16:59.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7f0d0e173640 1 -- 172.21.15.86:0/2163550721 learned_addr learned my addr 172.21.15.86:0/2163550721 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7f0d0e173640 10 monclient(hunting): get_auth_request con 0x7f0d080a8730 auth_method 0 2023-12-11T20:16:59.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7f0d0e173640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7f0d0e173640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7f0d0e173640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.209+0000 7fd1a6aeb640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fd184072c00 0x7fd184075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7fd196ffd640 10 client.4262.objecter ms_handle_connect 0x7fd184072c00 2023-12-11T20:16:59.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7fab8ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab9006f800 0x7fab901debd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9b1c0d2950 con 0x7f9b1402b2b0 2023-12-11T20:16:59.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 10 client.4278.objecter ms_dispatch 0x7f9b1c000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 3 client.4278.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 3 client.4278.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 10 client.4278.objecter checking command 1 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f9b00072c00 0x7f9b00075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 20 client.4278.objecter _get_session s=0x7f9b00072530 osd=3 3 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 20 client.4278.objecter put_session s=0x7f9b00072530 osd=3 3 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 15 client.4278.objecter _session_command_op_remove -1 1 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 20 client.4278.objecter _get_session s=0x7f9b00072530 osd=3 3 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 20 client.4278.objecter get_session s=0x7f9b00072530 osd=3 3 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 15 client.4278.objecter _session_command_op_assign 3 1 2023-12-11T20:16:59.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 20 client.4278.objecter put_session s=0x7f9b00072530 osd=3 4 2023-12-11T20:16:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 10 client.4278.objecter _send_command 1 2023-12-11T20:16:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 --> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b000756d0 con 0x7f9b00072c00 2023-12-11T20:16:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 20 client.4278.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b1c0cec90 con 0x7f9b1402b2b0 2023-12-11T20:16:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.210+0000 7f9b18ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:16:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7f0d0e173640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7f0d0e173640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7f0d0e173640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab9006f800 0x7fab901debd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40014/0 (socket says 172.21.15.86:40014) 2023-12-11T20:16:59.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 1 -- 172.21.15.86:0/2485929340 learned_addr learned my addr 172.21.15.86:0/2485929340 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 10 monclient(hunting): get_auth_request con 0x7fab9006f800 auth_method 0 2023-12-11T20:16:59.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7f9b20946640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f9b00072c00 0x7f9b00075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7f9b20946640 10 monclient: get_auth_request con 0x7f9b00072c00 auth_method 0 2023-12-11T20:16:59.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.211+0000 7fab8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.212+0000 7f0d0e974640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d081423a0 0x7f0d0813fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.213+0000 7f0d0e974640 10 monclient(hunting): get_auth_request con 0x7f0d081423a0 auth_method 0 2023-12-11T20:16:59.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.213+0000 7f0d0e974640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.213+0000 7f0d0e974640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.213+0000 7f0d0e974640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.214+0000 7f0d0e974640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.214+0000 7f0d0e974640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.214+0000 7f0d0e974640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 1 -- 172.21.15.86:0/3437017668 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa002f050 con 0x7f4aa8039190 2023-12-11T20:16:59.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f9b20946640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f9b00072c00 0x7f9b00075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2023-12-11T20:16:59.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:59.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f9b18ff9640 10 client.4278.objecter ms_handle_connect 0x7f9b00072c00 2023-12-11T20:16:59.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: dump: 2023-12-11T20:16:59.315 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.315 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.316 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.316 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.316 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.316 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.316 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.316 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.316 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.317 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 1 -- 172.21.15.86:0/3437017668 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0031700 con 0x7f4aa8039190 2023-12-11T20:16:59.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 1 -- 172.21.15.86:0/3437017668 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0031a90 con 0x7f4aa8039190 2023-12-11T20:16:59.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab4e7f640 10 monclient: dump: 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.318 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.319 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.319 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.319 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab6e83640 5 monclient: authenticate success, global_id 4284 2023-12-11T20:16:59.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab6e83640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f4ab6e83640 4 set_mon_vals no callback set 2023-12-11T20:16:59.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 1 -- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d0813ef70 msgr2=0x7f0d081375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d0813ef70 0x7f0d081375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 1 -- 172.21.15.86:0/2163550721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d081423a0 msgr2=0x7f0d0813fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d081423a0 0x7f0d0813fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.321 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.321 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.215+0000 7f0d0e173640 1 -- 172.21.15.86:0/2163550721 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80094d0 con 0x7f0d080a8730 2023-12-11T20:16:59.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 10 monclient: shutdown 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 1 -- 172.21.15.86:0/3437017668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8039190 msgr2=0x7f4aa80c8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 1 --2- 172.21.15.86:0/3437017668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8039190 0x7f4aa80c8350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097a0 tx=0x7f4aa002b7a0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 1 -- 172.21.15.86:0/3437017668 shutdown_connections 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 1 --2- 172.21.15.86:0/3437017668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80364c0 0x7f4aa80c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 1 --2- 172.21.15.86:0/3437017668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8039190 0x7f4aa80c8350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 1 -- 172.21.15.86:0/3437017668 >> 172.21.15.86:0/3437017668 conn(0x7f4aa8006610 msgr2=0x7f4aa8006a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 1 -- 172.21.15.86:0/3437017668 shutdown_connections 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.216+0000 7f4ab6e83640 1 -- 172.21.15.86:0/3437017668 wait complete. 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 10 monclient: monmap: 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.323 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 1 librados: starting msgr at 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 1 librados: starting objecter 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 1 Processor -- start 2023-12-11T20:16:59.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 1 -- start start 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 1 librados: setting wanted keys 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 1 librados: calling monclient init 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.217+0000 7f4ab6e83640 10 monclient: init 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4aa80364c0 0x7f4aa80ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa80364c0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8038db0 0x7f4aa80c1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa8038db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80c1990 0x7f4aa8160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa80c1990 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa80caa10 con 0x7f4aa80c1990 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa80cb920 con 0x7f4aa80364c0 2023-12-11T20:16:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa80c95a0 con 0x7f4aa8038db0 2023-12-11T20:16:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.218+0000 7f4ab6e83640 10 monclient(hunting): authenticate will time out at 1505.547545s 2023-12-11T20:16:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7f66c2ffd640 10 client.?.objecter ms_handle_connect 0x7f66cc0a8a20 2023-12-11T20:16:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7f66c2ffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7f66d2407640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66cc0a8a20 0x7f66cc136f90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66c400c7a0 tx=0x7f66c400cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f55a800cb00 con 0x7f55b414de80 2023-12-11T20:16:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7f55b0ff9640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f55a006c870 0x7f55a006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 1 -- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab900a93b0 msgr2=0x7fab90135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab900a93b0 0x7fab90135c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 1 -- 172.21.15.86:0/2485929340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab900aeae0 msgr2=0x7fab900a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab900aeae0 0x7fab900a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.219+0000 7fab8ffff640 1 -- 172.21.15.86:0/2485929340 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84008550 con 0x7fab9006f800 2023-12-11T20:16:59.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.220+0000 7f55b3fff640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f55a006c870 0x7f55a006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.220+0000 7f55b3fff640 10 monclient: get_auth_request con 0x7f55a006c870 auth_method 0 2023-12-11T20:16:59.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.220+0000 7f55b0ff9640 10 client.4351.objecter ms_handle_connect 0x7f55a006c870 2023-12-11T20:16:59.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.220+0000 7f55b3fff640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f55a006c870 0x7f55a006ed30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f55a403a000 tx=0x7f55a4034000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f4ab6682640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80c1990 0x7f4aa8160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f55a80588a0 con 0x7f55b414de80 2023-12-11T20:16:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 10 client.4351.objecter ms_dispatch 0x7f55b4000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 3 client.4351.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 3 client.4351.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 10 client.4351.objecter checking command 1 2023-12-11T20:16:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f55a0072ae0 0x7f55a0074f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 20 client.4351.objecter _get_session s=0x7f55a800eed0 osd=7 3 2023-12-11T20:16:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 20 client.4351.objecter put_session s=0x7f55a800eed0 osd=7 3 2023-12-11T20:16:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f4ab6682640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80c1990 0x7f4aa8160240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34886/0 (socket says 172.21.15.86:34886) 2023-12-11T20:16:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.223+0000 7f4ab6682640 1 -- 172.21.15.86:0/684777781 learned_addr learned my addr 172.21.15.86:0/684777781 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.223+0000 7f4ab6682640 10 monclient(hunting): get_auth_request con 0x7f4aa80c1990 auth_method 0 2023-12-11T20:16:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.223+0000 7f4ab6682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.223+0000 7f4ab6682640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.223+0000 7f4ab6682640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 15 client.4351.objecter _session_command_op_remove -1 1 2023-12-11T20:16:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 20 client.4351.objecter _get_session s=0x7f55a800eed0 osd=7 3 2023-12-11T20:16:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 20 client.4351.objecter get_session s=0x7f55a800eed0 osd=7 3 2023-12-11T20:16:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 15 client.4351.objecter _session_command_op_assign 7 1 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 20 client.4351.objecter put_session s=0x7f55a800eed0 osd=7 4 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 10 client.4351.objecter _send_command 1 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 --> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55a00755b0 con 0x7f55a0072ae0 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 20 client.4351.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55a80544e0 con 0x7f55b414de80 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.222+0000 7f55b0ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.223+0000 7f55b37fe640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f55a0072ae0 0x7f55a0074f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.223+0000 7f55b37fe640 10 monclient: get_auth_request con 0x7f55a0072ae0 auth_method 0 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7fab8ffff640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab9006f800 0x7fab901debd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab8800ba50 tx=0x7fab8800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7fab8cff9640 10 client.?.objecter ms_handle_connect 0x7fab9006f800 2023-12-11T20:16:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7fab8cff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7f4ab6682640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7f4ab6682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7f4ab6682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7f0d0e173640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d080a8730 0x7f0d08131660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400b4a0 tx=0x7f0d0400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7f0cfeffd640 10 client.?.objecter ms_handle_connect 0x7f0d080a8730 2023-12-11T20:16:59.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.224+0000 7f0cfeffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.226+0000 7f4ab5e81640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4aa80364c0 0x7f4aa80ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f55b37fe640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f55a0072ae0 0x7f55a0074f00 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f55b0ff9640 10 client.4351.objecter ms_handle_connect 0x7f55a0072ae0 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 1 -- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8038db0 msgr2=0x7f4aa80c1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8038db0 0x7f4aa80c1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 1 -- 172.21.15.86:0/684777781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4aa80364c0 msgr2=0x7f4aa80ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4aa80364c0 0x7f4aa80ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.227+0000 7f4ab6682640 1 -- 172.21.15.86:0/684777781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa0009400 con 0x7f4aa80c1990 2023-12-11T20:16:59.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.229+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 <== osd.5 v2:172.21.15.133:6802/3582621102 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9780075720 con 0x7f9780072c50 2023-12-11T20:16:59.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.229+0000 7f978effd640 10 client.4336.objecter ms_dispatch 0x7f9798000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:16:59.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.229+0000 7f978effd640 10 client.4336.objecter _finish_command 1 = system:0 2023-12-11T20:16:59.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.229+0000 7f978effd640 20 client.4336.objecter put_session s=0x7f9780072580 osd=5 4 2023-12-11T20:16:59.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.229+0000 7f978effd640 15 client.4336.objecter _session_command_op_remove 5 1 2023-12-11T20:16:59.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f4ab6682640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80c1990 0x7f4aa8160240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0068520 tx=0x7f4ab0067960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f4aa67fc640 10 client.?.objecter ms_handle_connect 0x7f4aa80c1990 2023-12-11T20:16:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f4aa67fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4003d80 con 0x7f66cc0a8a20 2023-12-11T20:16:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66c2ffd640 10 monclient: dump: 2023-12-11T20:16:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66c2ffd640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66c2ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66d466b640 5 monclient: authenticate success, global_id 4354 2023-12-11T20:16:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66d466b640 10 monclient: _renew_subs 2023-12-11T20:16:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66d466b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.230+0000 7f66d466b640 1 -- 172.21.15.86:0/1220750924 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc1366e0 con 0x7f66cc0a8a20 2023-12-11T20:16:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c4005070 con 0x7f66cc0a8a20 2023-12-11T20:16:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66c2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66d466b640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66d466b640 10 monclient: _renew_subs 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66d466b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66d466b640 1 -- 172.21.15.86:0/1220750924 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc1418f0 con 0x7f66cc0a8a20 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66d466b640 1 librados: init done 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4003d80 con 0x7f66cc0a8a20 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66c2ffd640 10 monclient: dump: 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.342 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.231+0000 7f66d0c04640 4 set_mon_vals no callback set 2023-12-11T20:16:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7f66c0ff9640 10 client.4354.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7f66c0ff9640 15 client.4354.objecter _session_command_op_assign -1 1 2023-12-11T20:16:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7f66c0ff9640 20 client.4354.objecter _get_session osd=-1 returning homeless 2023-12-11T20:16:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7f66c0ff9640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7f66c0ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66a8002910 con 0x7f66cc0a8a20 2023-12-11T20:16:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8800cbe0 con 0x7fab9006f800 2023-12-11T20:16:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: dump: 2023-12-11T20:16:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.347 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8800cd60 con 0x7fab9006f800 2023-12-11T20:16:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8e7fc640 4 set_mon_vals no callback set 2023-12-11T20:16:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88014680 con 0x7fab9006f800 2023-12-11T20:16:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab8cff9640 10 monclient: dump: 2023-12-11T20:16:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab96973640 5 monclient: authenticate success, global_id 4277 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab96973640 10 monclient: _renew_subs 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab96973640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9013bba0 con 0x7fab9006f800 2023-12-11T20:16:59.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab96973640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab96973640 10 monclient: _renew_subs 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab96973640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.232+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab901df110 con 0x7fab9006f800 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.233+0000 7fab96973640 1 librados: init done 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.233+0000 7fab96973640 10 client.4277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.233+0000 7fab96973640 15 client.4277.objecter _session_command_op_assign -1 1 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.233+0000 7fab96973640 20 client.4277.objecter _get_session osd=-1 returning homeless 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.233+0000 7fab96973640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.233+0000 7fab96973640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.233+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab9013ba20 con 0x7fab9006f800 2023-12-11T20:16:59.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab008f070 con 0x7f4aa80c1990 2023-12-11T20:16:59.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:59.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 10 monclient: dump: 2023-12-11T20:16:59.352 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.352 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab006ccf0 con 0x7f4aa80c1990 2023-12-11T20:16:59.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4ab6e83640 5 monclient: authenticate success, global_id 4293 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04010610 con 0x7f0d080a8730 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: dump: 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.355 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4ab6e83640 10 monclient: _renew_subs 2023-12-11T20:16:59.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d04010770 con 0x7f0d080a8730 2023-12-11T20:16:59.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4ab6e83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4ab6e83640 1 -- 172.21.15.86:0/684777781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa80ca550 con 0x7f4aa80c1990 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04010610 con 0x7f0d080a8730 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0d0f175640 5 monclient: authenticate success, global_id 4274 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0cfeffd640 10 monclient: dump: 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0d0f175640 10 monclient: _renew_subs 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0d0c970640 4 set_mon_vals no callback set 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0d0f175640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d08140090 con 0x7f0d080a8730 2023-12-11T20:16:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4ab6e83640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa67fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0d0f175640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0d0f175640 10 monclient: _renew_subs 2023-12-11T20:16:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f0d0f175640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d080b7710 con 0x7f0d080a8730 2023-12-11T20:16:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f0d0f175640 1 librados: init done 2023-12-11T20:16:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.234+0000 7f4aa7fff640 4 set_mon_vals no callback set 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0072350 con 0x7f4aa80c1990 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f4aa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f4aa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f4aa67fc640 10 monclient: dump: 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.361 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f4ab6e83640 10 monclient: _renew_subs 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f4ab6e83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f4ab6e83640 1 -- 172.21.15.86:0/684777781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa80c2130 con 0x7f4aa80c1990 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.235+0000 7f4ab6e83640 1 librados: init done 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.236+0000 7f0d0f175640 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.236+0000 7f0d0f175640 15 client.4274.objecter _session_command_op_assign -1 1 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.236+0000 7f0d0f175640 20 client.4274.objecter _get_session osd=-1 returning homeless 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.236+0000 7f0d0f175640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.236+0000 7f0d0f175640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.236+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d08141e80 con 0x7f0d080a8730 2023-12-11T20:16:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.236+0000 7f4a93fff640 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.237+0000 7f4a93fff640 15 client.4293.objecter _session_command_op_assign -1 1 2023-12-11T20:16:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.237+0000 7f4a93fff640 20 client.4293.objecter _get_session osd=-1 returning homeless 2023-12-11T20:16:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.237+0000 7f4a93fff640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.237+0000 7f4a93fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.237+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4a8c002910 con 0x7f4aa80c1990 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.252+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f66c4005220 con 0x7f66cc0a8a20 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.252+0000 7f66c2ffd640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f66b006c8e0 0x7f66b006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.253+0000 7f66d1c06640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f66b006c8e0 0x7f66b006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.253+0000 7f66d1c06640 10 monclient: get_auth_request con 0x7f66b006c8e0 auth_method 0 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.253+0000 7f66c2ffd640 10 client.4354.objecter ms_handle_connect 0x7f66b006c8e0 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.253+0000 7f66d1c06640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f66b006c8e0 0x7f66b006eda0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f66c8036000 tx=0x7f66c8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f66c4058330 con 0x7f66cc0a8a20 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 10 client.4354.objecter ms_dispatch 0x7f66cc000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 3 client.4354.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 3 client.4354.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 10 client.4354.objecter checking command 1 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f66b0072cc0 0x7f66b00750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 20 client.4354.objecter _get_session s=0x7f66b00725f0 osd=1 3 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 20 client.4354.objecter put_session s=0x7f66b00725f0 osd=1 3 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 15 client.4354.objecter _session_command_op_remove -1 1 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 20 client.4354.objecter _get_session s=0x7f66b00725f0 osd=1 3 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 20 client.4354.objecter get_session s=0x7f66b00725f0 osd=1 3 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 15 client.4354.objecter _session_command_op_assign 1 1 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 20 client.4354.objecter put_session s=0x7f66b00725f0 osd=1 4 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 10 client.4354.objecter _send_command 1 2023-12-11T20:16:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 --> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b0075790 con 0x7f66b0072cc0 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 20 client.4354.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66c4003d80 con 0x7f66cc0a8a20 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.254+0000 7f66c2ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.255+0000 7f66d2c08640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f66b0072cc0 0x7f66b00750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.255+0000 7f66d2c08640 10 monclient: get_auth_request con 0x7f66b0072cc0 auth_method 0 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.259+0000 7f66d2c08640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f66b0072cc0 0x7f66b00750e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.259+0000 7f66c2ffd640 10 client.4354.objecter ms_handle_connect 0x7f66b0072cc0 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 10 client.4336.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-11T20:16:59.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 15 client.4336.objecter _session_command_op_assign -1 2 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 20 client.4336.objecter _get_session s=0x7f9780072580 osd=5 3 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 20 client.4336.objecter put_session s=0x7f9780072580 osd=5 3 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 20 client.4336.objecter _get_session s=0x7f9780072580 osd=5 3 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 15 client.4336.objecter _session_command_op_remove -1 2 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 20 client.4336.objecter get_session s=0x7f9780072580 osd=5 3 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 15 client.4336.objecter _session_command_op_assign 5 2 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 20 client.4336.objecter put_session s=0x7f9780072580 osd=5 4 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 10 client.4336.objecter _send_command 2 2023-12-11T20:16:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.260+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 --> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9774001150 con 0x7f9780072c50 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.262+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4ab00724b0 con 0x7f4aa80c1990 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.262+0000 7f4aa67fc640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4a9406c820 0x7f4a9406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.263+0000 7f4ab5e81640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4a9406c820 0x7f4a9406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.263+0000 7f4ab5e81640 10 monclient: get_auth_request con 0x7f4a9406c820 auth_method 0 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.263+0000 7f4ab5e81640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4a9406c820 0x7f4a9406ece0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0015d0 tx=0x7f4aac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.263+0000 7f4aa67fc640 10 client.4293.objecter ms_handle_connect 0x7f4a9406c820 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4ab00d2e80 con 0x7f4aa80c1990 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 10 client.4293.objecter ms_dispatch 0x7f4ab0000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 3 client.4293.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 3 client.4293.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 10 client.4293.objecter checking command 1 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f4a94072c00 0x7f4a94075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 20 client.4293.objecter _get_session s=0x7f4a94072530 osd=0 3 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 20 client.4293.objecter put_session s=0x7f4a94072530 osd=0 3 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 15 client.4293.objecter _session_command_op_remove -1 1 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 20 client.4293.objecter _get_session s=0x7f4a94072530 osd=0 3 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 20 client.4293.objecter get_session s=0x7f4a94072530 osd=0 3 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 15 client.4293.objecter _session_command_op_assign 0 1 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 20 client.4293.objecter put_session s=0x7f4a94072530 osd=0 4 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 10 client.4293.objecter _send_command 1 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 --> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a940756d0 con 0x7f4a94072c00 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 20 client.4293.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ab00ceac0 con 0x7f4aa80c1990 2023-12-11T20:16:59.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.265+0000 7f4aa67fc640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:16:59.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.266+0000 7f4ab5680640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f4a94072c00 0x7f4a94075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.266+0000 7f4ab5680640 10 monclient: get_auth_request con 0x7f4a94072c00 auth_method 0 2023-12-11T20:16:59.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.266+0000 7f978effd640 1 -- 172.21.15.86:0/517550087 <== osd.5 v2:172.21.15.133:6802/3582621102 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9774001150 con 0x7f9780072c50 2023-12-11T20:16:59.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.266+0000 7f978effd640 10 client.4336.objecter ms_dispatch 0x7f9798000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:16:59.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.266+0000 7f978effd640 10 client.4336.objecter _finish_command 2 = system:0 2023-12-11T20:16:59.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.266+0000 7f978effd640 20 client.4336.objecter put_session s=0x7f9780072580 osd=5 4 2023-12-11T20:16:59.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.266+0000 7f978effd640 15 client.4336.objecter _session_command_op_remove 5 2 2023-12-11T20:16:59.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 10 librados: watch_flush enter 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 10 librados: watch_flush exit 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 10 client.4336.objecter close_session for osd.5 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f9780072c50 msgr2=0x7f9780075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f9780072c50 0x7f9780075070 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 20 client.4336.objecter put_session s=0x7f9780072580 osd=5 1 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 20 client.4336.objecter shutdown clearing up homeless session... 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 10 client.4336.objecter successfully canceled tick 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f978006c870 msgr2=0x7f978006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f978006c870 0x7f978006ed30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9784006a90 tx=0x7f9784006a20 comp rx=0 tx=0).stop 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 10 monclient: shutdown 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f979806f000 msgr2=0x7f97980ad1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 --2- 172.21.15.86:0/517550087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f979806f000 0x7f97980ad1e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f978800e870 tx=0x7f978800ed30 comp rx=0 tx=0).stop 2023-12-11T20:16:59.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f9796d9d640 1 -- 172.21.15.86:0/517550087 reap_dead start 2023-12-11T20:16:59.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 shutdown_connections 2023-12-11T20:16:59.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 >> 172.21.15.86:0/517550087 conn(0x7f979806af10 msgr2=0x7f97980ae820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 shutdown_connections 2023-12-11T20:16:59.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 -- 172.21.15.86:0/517550087 wait complete. 2023-12-11T20:16:59.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.267+0000 7f979cd90640 1 librados: shutdown 2023-12-11T20:16:59.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.270+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0d040108e0 con 0x7f0d080a8730 2023-12-11T20:16:59.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.271+0000 7f0cfeffd640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0cec06c880 0x7f0cec06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.271+0000 7f0d0d972640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0cec06c880 0x7f0cec06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.271+0000 7f0d0d972640 10 monclient: get_auth_request con 0x7f0cec06c880 auth_method 0 2023-12-11T20:16:59.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.272+0000 7f0d0d972640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0cec06c880 0x7f0cec06ed40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8035000 tx=0x7f0cf8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.272+0000 7f0cfeffd640 10 client.4274.objecter ms_handle_connect 0x7f0cec06c880 2023-12-11T20:16:59.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.272+0000 7f4aa67fc640 10 client.4293.objecter ms_handle_connect 0x7f4a94072c00 2023-12-11T20:16:59.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.272+0000 7f4ab5680640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f4a94072c00 0x7f4a94075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.273+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 <== osd.3 v2:172.21.15.86:6800/4064445038 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9b000756d0 con 0x7f9b00072c00 2023-12-11T20:16:59.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.273+0000 7f9b18ff9640 10 client.4278.objecter ms_dispatch 0x7f9b1c000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.273+0000 7f9b18ff9640 10 client.4278.objecter _finish_command 1 = system:0 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.273+0000 7f9b18ff9640 20 client.4278.objecter put_session s=0x7f9b00072530 osd=3 4 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.273+0000 7f9b18ff9640 15 client.4278.objecter _session_command_op_remove 3 1 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0d04058180 con 0x7f0d080a8730 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 10 client.4274.objecter ms_dispatch 0x7f0d08000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 3 client.4274.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 3 client.4274.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 10 client.4274.objecter checking command 1 2023-12-11T20:16:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f0cec073580 0x7f0cec075a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 20 client.4274.objecter _get_session s=0x7f0cec072eb0 osd=4 3 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 20 client.4274.objecter put_session s=0x7f0cec072eb0 osd=4 3 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 15 client.4274.objecter _session_command_op_remove -1 1 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 20 client.4274.objecter _get_session s=0x7f0cec072eb0 osd=4 3 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 20 client.4274.objecter get_session s=0x7f0cec072eb0 osd=4 3 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 15 client.4274.objecter _session_command_op_assign 4 1 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 20 client.4274.objecter put_session s=0x7f0cec072eb0 osd=4 4 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 10 client.4274.objecter _send_command 1 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 --> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cec0760f0 con 0x7f0cec073580 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f0cfeffd640 20 client.4274.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 <== osd.7 v2:172.21.15.133:6810/1136721322 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f55a00755b0 con 0x7f55a0072ae0 2023-12-11T20:16:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f55b0ff9640 10 client.4351.objecter ms_dispatch 0x7f55b4000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:16:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f55b0ff9640 10 client.4351.objecter _finish_command 1 = system:0 2023-12-11T20:16:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f55b0ff9640 20 client.4351.objecter put_session s=0x7f55a800eed0 osd=7 4 2023-12-11T20:16:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.274+0000 7f55b0ff9640 15 client.4351.objecter _session_command_op_remove 7 1 2023-12-11T20:16:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.275+0000 7f0d0e974640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f0cec073580 0x7f0cec075a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.275+0000 7f0d0e974640 10 monclient: get_auth_request con 0x7f0cec073580 auth_method 0 2023-12-11T20:16:59.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.278+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d040092d0 con 0x7f0d080a8730 2023-12-11T20:16:59.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.278+0000 7f0cfeffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:16:59.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.279+0000 7f0d0e974640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f0cec073580 0x7f0cec075a20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.279+0000 7f0cfeffd640 10 client.4274.objecter ms_handle_connect 0x7f0cec073580 2023-12-11T20:16:59.383 INFO:teuthology.orchestra.run.smithi086.stdout:55834574853 2023-12-11T20:16:59.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-11T20:16:59.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.285+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 <== osd.2 v2:172.21.15.86:6816/406445159 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd1840756d0 con 0x7fd184072c00 2023-12-11T20:16:59.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.285+0000 7fd196ffd640 10 client.4262.objecter ms_dispatch 0x7fd1a0000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:16:59.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.285+0000 7fd196ffd640 10 client.4262.objecter _finish_command 1 = system:0 2023-12-11T20:16:59.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.285+0000 7fd196ffd640 20 client.4262.objecter put_session s=0x7fd184072530 osd=2 4 2023-12-11T20:16:59.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.285+0000 7fd196ffd640 15 client.4262.objecter _session_command_op_remove 2 1 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 10 client.4351.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 15 client.4351.objecter _session_command_op_assign -1 2 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 20 client.4351.objecter _get_session s=0x7f55a800eed0 osd=7 3 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 20 client.4351.objecter put_session s=0x7f55a800eed0 osd=7 3 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 20 client.4351.objecter _get_session s=0x7f55a800eed0 osd=7 3 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 15 client.4351.objecter _session_command_op_remove -1 2 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 20 client.4351.objecter get_session s=0x7f55a800eed0 osd=7 3 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 15 client.4351.objecter _session_command_op_assign 7 2 2023-12-11T20:16:59.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 20 client.4351.objecter put_session s=0x7f55a800eed0 osd=7 4 2023-12-11T20:16:59.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 10 client.4351.objecter _send_command 2 2023-12-11T20:16:59.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fab88007370 con 0x7fab9006f800 2023-12-11T20:16:59.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.295+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 --> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f55b40adb90 con 0x7f55a0072ae0 2023-12-11T20:16:59.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.296+0000 7fab8cff9640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fab7c06c870 0x7fab7c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.296+0000 7fab8f7fe640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fab7c06c870 0x7fab7c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.296+0000 7fab8f7fe640 10 monclient: get_auth_request con 0x7fab7c06c870 auth_method 0 2023-12-11T20:16:59.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.296+0000 7fab8f7fe640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fab7c06c870 0x7fab7c06ed30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab8403b000 tx=0x7fab84008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.296+0000 7fab8cff9640 10 client.4277.objecter ms_handle_connect 0x7fab7c06c870 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fab880591f0 con 0x7fab9006f800 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 10 client.4277.objecter ms_dispatch 0x7fab90000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 3 client.4277.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 3 client.4277.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 10 client.4277.objecter checking command 1 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7fab7c072c50 0x7fab7c075070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 20 client.4277.objecter _get_session s=0x7fab7c072580 osd=6 3 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 20 client.4277.objecter put_session s=0x7fab7c072580 osd=6 3 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 15 client.4277.objecter _session_command_op_remove -1 1 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 20 client.4277.objecter _get_session s=0x7fab7c072580 osd=6 3 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 20 client.4277.objecter get_session s=0x7fab7c072580 osd=6 3 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 15 client.4277.objecter _session_command_op_assign 6 1 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 20 client.4277.objecter put_session s=0x7fab7c072580 osd=6 4 2023-12-11T20:16:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.298+0000 7fab8cff9640 10 client.4277.objecter _send_command 1 2023-12-11T20:16:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab7c075720 con 0x7fab7c072c50 2023-12-11T20:16:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7fab8cff9640 20 client.4277.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab88054e30 con 0x7fab9006f800 2023-12-11T20:16:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7fab8cff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:16:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7fab94f10640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7fab7c072c50 0x7fab7c075070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 10 client.4278.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-11T20:16:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 15 client.4278.objecter _session_command_op_assign -1 2 2023-12-11T20:16:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 20 client.4278.objecter _get_session s=0x7f9b00072530 osd=3 3 2023-12-11T20:16:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 20 client.4278.objecter put_session s=0x7f9b00072530 osd=3 3 2023-12-11T20:16:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 20 client.4278.objecter _get_session s=0x7f9b00072530 osd=3 3 2023-12-11T20:16:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 15 client.4278.objecter _session_command_op_remove -1 2 2023-12-11T20:16:59.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 20 client.4278.objecter get_session s=0x7f9b00072530 osd=3 3 2023-12-11T20:16:59.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 15 client.4278.objecter _session_command_op_assign 3 2 2023-12-11T20:16:59.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 20 client.4278.objecter put_session s=0x7f9b00072530 osd=3 4 2023-12-11T20:16:59.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 10 client.4278.objecter _send_command 2 2023-12-11T20:16:59.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.299+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 --> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9b14033a90 con 0x7f9b00072c00 2023-12-11T20:16:59.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.300+0000 7fab94f10640 10 monclient: get_auth_request con 0x7fab7c072c50 auth_method 0 2023-12-11T20:16:59.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.303+0000 7f55b0ff9640 1 -- 172.21.15.86:0/1073111984 <== osd.7 v2:172.21.15.133:6810/1136721322 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f55ac003fe0 con 0x7f55a0072ae0 2023-12-11T20:16:59.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.303+0000 7f55b0ff9640 10 client.4351.objecter ms_dispatch 0x7f55b4000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:16:59.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.303+0000 7f55b0ff9640 10 client.4351.objecter _finish_command 2 = system:0 2023-12-11T20:16:59.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.303+0000 7f55b0ff9640 20 client.4351.objecter put_session s=0x7f55a800eed0 osd=7 4 2023-12-11T20:16:59.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.303+0000 7f55b0ff9640 15 client.4351.objecter _session_command_op_remove 7 2 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 10 librados: watch_flush enter 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 10 librados: watch_flush exit 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 10 client.4351.objecter close_session for osd.7 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f55a0072ae0 msgr2=0x7f55a0074f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f55a0072ae0 0x7f55a0074f00 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 20 client.4351.objecter put_session s=0x7f55a800eed0 osd=7 1 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 20 client.4351.objecter shutdown clearing up homeless session... 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 10 client.4351.objecter successfully canceled tick 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f55a006c870 msgr2=0x7f55a006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f55a006c870 0x7f55a006ed30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f55a403a000 tx=0x7f55a4034000 comp rx=0 tx=0).stop 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 10 monclient: shutdown 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f55b414de80 msgr2=0x7f55b414e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 --2- 172.21.15.86:0/1073111984 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f55b414de80 0x7f55b414e300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f55b4071570 tx=0x7f55a800eb90 comp rx=0 tx=0).stop 2023-12-11T20:16:59.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55b8de8640 1 -- 172.21.15.86:0/1073111984 reap_dead start 2023-12-11T20:16:59.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 shutdown_connections 2023-12-11T20:16:59.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 >> 172.21.15.86:0/1073111984 conn(0x7f55b406a6e0 msgr2=0x7f55b40ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 shutdown_connections 2023-12-11T20:16:59.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 -- 172.21.15.86:0/1073111984 wait complete. 2023-12-11T20:16:59.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.304+0000 7f55ba84b640 1 librados: shutdown 2023-12-11T20:16:59.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.305+0000 7fab94f10640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7fab7c072c50 0x7fab7c075070 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.305+0000 7fab8cff9640 10 client.4277.objecter ms_handle_connect 0x7fab7c072c50 2023-12-11T20:16:59.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.307+0000 7f9b18ff9640 1 -- 172.21.15.86:0/620246898 <== osd.3 v2:172.21.15.86:6800/4064445038 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9b10031a40 con 0x7f9b00072c00 2023-12-11T20:16:59.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.307+0000 7f9b18ff9640 10 client.4278.objecter ms_dispatch 0x7f9b1c000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:16:59.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.307+0000 7f9b18ff9640 10 client.4278.objecter _finish_command 2 = system:0 2023-12-11T20:16:59.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.307+0000 7f9b18ff9640 20 client.4278.objecter put_session s=0x7f9b00072530 osd=3 4 2023-12-11T20:16:59.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.307+0000 7f9b18ff9640 15 client.4278.objecter _session_command_op_remove 3 2 2023-12-11T20:16:59.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 10 librados: watch_flush enter 2023-12-11T20:16:59.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 10 librados: watch_flush exit 2023-12-11T20:16:59.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 10 client.4278.objecter close_session for osd.3 2023-12-11T20:16:59.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f9b00072c00 msgr2=0x7f9b00075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f9b00072c00 0x7f9b00075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 20 client.4278.objecter put_session s=0x7f9b00072530 osd=3 1 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 20 client.4278.objecter shutdown clearing up homeless session... 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 10 client.4278.objecter successfully canceled tick 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9b0006c820 msgr2=0x7f9b0006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9b0006c820 0x7f9b0006ece0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9b08009a00 tx=0x7f9b08009190 comp rx=0 tx=0).stop 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 10 monclient: shutdown 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b1402b2b0 msgr2=0x7f9b1415cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 --2- 172.21.15.86:0/620246898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b1402b2b0 0x7f9b1415cc00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c073030 tx=0x7f9b1c067960 comp rx=0 tx=0).stop 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b20946640 1 -- 172.21.15.86:0/620246898 reap_dead start 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 shutdown_connections 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 >> 172.21.15.86:0/620246898 conn(0x7f9b14006510 msgr2=0x7f9b140c0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 shutdown_connections 2023-12-11T20:16:59.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 -- 172.21.15.86:0/620246898 wait complete. 2023-12-11T20:16:59.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.308+0000 7f9b21147640 1 librados: shutdown 2023-12-11T20:16:59.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.317+0000 7fd1a72ec640 10 client.4262.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-11T20:16:59.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 15 client.4262.objecter _session_command_op_assign -1 2 2023-12-11T20:16:59.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 20 client.4262.objecter _get_session s=0x7fd184072530 osd=2 3 2023-12-11T20:16:59.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 20 client.4262.objecter put_session s=0x7fd184072530 osd=2 3 2023-12-11T20:16:59.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 20 client.4262.objecter _get_session s=0x7fd184072530 osd=2 3 2023-12-11T20:16:59.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 15 client.4262.objecter _session_command_op_remove -1 2 2023-12-11T20:16:59.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 20 client.4262.objecter get_session s=0x7fd184072530 osd=2 3 2023-12-11T20:16:59.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 15 client.4262.objecter _session_command_op_assign 2 2 2023-12-11T20:16:59.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 20 client.4262.objecter put_session s=0x7fd184072530 osd=2 4 2023-12-11T20:16:59.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 10 client.4262.objecter _send_command 2 2023-12-11T20:16:59.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.318+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 --> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd188001240 con 0x7fd184072c00 2023-12-11T20:16:59.402 INFO:teuthology.orchestra.run.smithi086.stdout:55834574854 2023-12-11T20:16:59.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-11T20:16:59.403 INFO:teuthology.orchestra.run.smithi086.stdout:68719476743 2023-12-11T20:16:59.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-11T20:16:59.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.325+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 <== osd.1 v2:172.21.15.86:6808/4202858331 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f66b0075790 con 0x7f66b0072cc0 2023-12-11T20:16:59.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.325+0000 7f66c2ffd640 10 client.4354.objecter ms_dispatch 0x7f66cc000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:16:59.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.325+0000 7f66c2ffd640 10 client.4354.objecter _finish_command 1 = system:0 2023-12-11T20:16:59.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.325+0000 7f66c2ffd640 20 client.4354.objecter put_session s=0x7f66b00725f0 osd=1 4 2023-12-11T20:16:59.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.325+0000 7f66c2ffd640 15 client.4354.objecter _session_command_op_remove 1 1 2023-12-11T20:16:59.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7fd196ffd640 1 -- 172.21.15.86:0/2256697928 <== osd.2 v2:172.21.15.86:6816/406445159 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd188001240 con 0x7fd184072c00 2023-12-11T20:16:59.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7fd196ffd640 10 client.4262.objecter ms_dispatch 0x7fd1a0000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:16:59.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7fd196ffd640 10 client.4262.objecter _finish_command 2 = system:0 2023-12-11T20:16:59.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7fd196ffd640 20 client.4262.objecter put_session s=0x7fd184072530 osd=2 4 2023-12-11T20:16:59.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7fd196ffd640 15 client.4262.objecter _session_command_op_remove 2 2 2023-12-11T20:16:59.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 <== osd.4 v2:172.21.15.133:6818/1433273204 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0cec0760f0 con 0x7f0cec073580 2023-12-11T20:16:59.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7f0cfeffd640 10 client.4274.objecter ms_dispatch 0x7f0d08000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:16:59.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7f0cfeffd640 10 client.4274.objecter _finish_command 1 = system:0 2023-12-11T20:16:59.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7f0cfeffd640 20 client.4274.objecter put_session s=0x7f0cec072eb0 osd=4 4 2023-12-11T20:16:59.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.328+0000 7f0cfeffd640 15 client.4274.objecter _session_command_op_remove 4 1 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 10 librados: watch_flush enter 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 10 librados: watch_flush exit 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 10 client.4262.objecter close_session for osd.2 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fd184072c00 msgr2=0x7fd184075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fd184072c00 0x7fd184075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 20 client.4262.objecter put_session s=0x7fd184072530 osd=2 1 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 20 client.4262.objecter shutdown clearing up homeless session... 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 10 client.4262.objecter successfully canceled tick 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd18406c820 msgr2=0x7fd18406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd18406c820 0x7fd18406ece0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd19800ab90 tx=0x7fd198009250 comp rx=0 tx=0).stop 2023-12-11T20:16:59.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 10 monclient: shutdown 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a006f1b0 msgr2=0x7fd1a00aafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 --2- 172.21.15.86:0/2256697928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1a006f1b0 0x7fd1a00aafd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c004730 comp rx=0 tx=0).stop 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a6aeb640 1 -- 172.21.15.86:0/2256697928 reap_dead start 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 shutdown_connections 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 >> 172.21.15.86:0/2256697928 conn(0x7fd1a006a500 msgr2=0x7fd1a006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.329+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 shutdown_connections 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.330+0000 7fd1a72ec640 1 -- 172.21.15.86:0/2256697928 wait complete. 2023-12-11T20:16:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.330+0000 7fd1a72ec640 1 librados: shutdown 2023-12-11T20:16:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.336+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 <== osd.0 v2:172.21.15.86:6820/1914311689 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4a940756d0 con 0x7f4a94072c00 2023-12-11T20:16:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.336+0000 7f4aa67fc640 10 client.4293.objecter ms_dispatch 0x7f4ab0000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:16:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.336+0000 7f4aa67fc640 10 client.4293.objecter _finish_command 1 = system:0 2023-12-11T20:16:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.336+0000 7f4aa67fc640 20 client.4293.objecter put_session s=0x7f4a94072530 osd=0 4 2023-12-11T20:16:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.336+0000 7f4aa67fc640 15 client.4293.objecter _session_command_op_remove 0 1 2023-12-11T20:16:59.409 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:16:59.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-11T20:16:59.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-11T20:16:59.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 15 client.4274.objecter _session_command_op_assign -1 2 2023-12-11T20:16:59.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 20 client.4274.objecter _get_session s=0x7f0cec072eb0 osd=4 3 2023-12-11T20:16:59.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 20 client.4274.objecter put_session s=0x7f0cec072eb0 osd=4 3 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 20 client.4274.objecter _get_session s=0x7f0cec072eb0 osd=4 3 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 15 client.4274.objecter _session_command_op_remove -1 2 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 20 client.4274.objecter get_session s=0x7f0cec072eb0 osd=4 3 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 15 client.4274.objecter _session_command_op_assign 4 2 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 20 client.4274.objecter put_session s=0x7f0cec072eb0 osd=4 4 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 10 client.4274.objecter _send_command 2 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.346+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 --> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d08138cb0 con 0x7f0cec073580 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 10 client.4354.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 15 client.4354.objecter _session_command_op_assign -1 2 2023-12-11T20:16:59.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 20 client.4354.objecter _get_session s=0x7f66b00725f0 osd=1 3 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 20 client.4354.objecter put_session s=0x7f66b00725f0 osd=1 3 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 20 client.4354.objecter _get_session s=0x7f66b00725f0 osd=1 3 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 15 client.4354.objecter _session_command_op_remove -1 2 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 20 client.4354.objecter get_session s=0x7f66b00725f0 osd=1 3 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 15 client.4354.objecter _session_command_op_assign 1 2 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 20 client.4354.objecter put_session s=0x7f66b00725f0 osd=1 4 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 10 client.4354.objecter _send_command 2 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.347+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 --> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f66a8001240 con 0x7f66b0072cc0 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.354+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 <== osd.6 v2:172.21.15.133:6826/1611390427 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fab7c075720 con 0x7fab7c072c50 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.354+0000 7fab8cff9640 10 client.4277.objecter ms_dispatch 0x7fab90000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:16:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.354+0000 7fab8cff9640 10 client.4277.objecter _finish_command 1 = system:0 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.354+0000 7fab8cff9640 20 client.4277.objecter put_session s=0x7fab7c072580 osd=6 4 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.354+0000 7fab8cff9640 15 client.4277.objecter _session_command_op_remove 6 1 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 15 client.4293.objecter _session_command_op_assign -1 2 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 20 client.4293.objecter _get_session s=0x7f4a94072530 osd=0 3 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 20 client.4293.objecter put_session s=0x7f4a94072530 osd=0 3 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 20 client.4293.objecter _get_session s=0x7f4a94072530 osd=0 3 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 15 client.4293.objecter _session_command_op_remove -1 2 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 20 client.4293.objecter get_session s=0x7f4a94072530 osd=0 3 2023-12-11T20:16:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 15 client.4293.objecter _session_command_op_assign 0 2 2023-12-11T20:16:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 20 client.4293.objecter put_session s=0x7f4a94072530 osd=0 4 2023-12-11T20:16:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 10 client.4293.objecter _send_command 2 2023-12-11T20:16:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.355+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 --> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4a8c001240 con 0x7f4a94072c00 2023-12-11T20:16:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c2ffd640 1 -- 172.21.15.86:0/1220750924 <== osd.1 v2:172.21.15.86:6808/4202858331 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f66a8001240 con 0x7f66b0072cc0 2023-12-11T20:16:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c2ffd640 10 client.4354.objecter ms_dispatch 0x7f66cc000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:16:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c2ffd640 10 client.4354.objecter _finish_command 2 = system:0 2023-12-11T20:16:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c2ffd640 20 client.4354.objecter put_session s=0x7f66b00725f0 osd=1 4 2023-12-11T20:16:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c2ffd640 15 client.4354.objecter _session_command_op_remove 1 2 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 10 librados: watch_flush enter 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 10 librados: watch_flush exit 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 10 client.4354.objecter close_session for osd.1 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f66b0072cc0 msgr2=0x7f66b00750e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f66b0072cc0 0x7f66b00750e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 20 client.4354.objecter put_session s=0x7f66b00725f0 osd=1 1 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 20 client.4354.objecter shutdown clearing up homeless session... 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 10 client.4354.objecter successfully canceled tick 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f66b006c8e0 msgr2=0x7f66b006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f66b006c8e0 0x7f66b006eda0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f66c8036000 tx=0x7f66c8030000 comp rx=0 tx=0).stop 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 10 monclient: shutdown 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66cc0a8a20 msgr2=0x7f66cc136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66c0ff9640 1 --2- 172.21.15.86:0/1220750924 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66cc0a8a20 0x7f66cc136f90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66c400c7a0 tx=0x7f66c400cc60 comp rx=0 tx=0).stop 2023-12-11T20:16:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.356+0000 7f66d2c08640 1 -- 172.21.15.86:0/1220750924 reap_dead start 2023-12-11T20:16:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.357+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 shutdown_connections 2023-12-11T20:16:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.357+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 >> 172.21.15.86:0/1220750924 conn(0x7f66cc0b7890 msgr2=0x7f66cc13e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.357+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 shutdown_connections 2023-12-11T20:16:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.357+0000 7f66c0ff9640 1 -- 172.21.15.86:0/1220750924 wait complete. 2023-12-11T20:16:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.357+0000 7f66c0ff9640 1 librados: shutdown 2023-12-11T20:16:59.417 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:16:59.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.372+0000 7fab96973640 10 client.4277.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.372+0000 7fab96973640 15 client.4277.objecter _session_command_op_assign -1 2 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.372+0000 7fab96973640 20 client.4277.objecter _get_session s=0x7fab7c072580 osd=6 3 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.373+0000 7fab96973640 20 client.4277.objecter put_session s=0x7fab7c072580 osd=6 3 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.373+0000 7fab96973640 20 client.4277.objecter _get_session s=0x7fab7c072580 osd=6 3 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.373+0000 7fab96973640 15 client.4277.objecter _session_command_op_remove -1 2 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.373+0000 7fab96973640 20 client.4277.objecter get_session s=0x7fab7c072580 osd=6 3 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.373+0000 7fab96973640 15 client.4277.objecter _session_command_op_assign 6 2 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.373+0000 7fab96973640 20 client.4277.objecter put_session s=0x7fab7c072580 osd=6 4 2023-12-11T20:16:59.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.373+0000 7fab96973640 10 client.4277.objecter _send_command 2 2023-12-11T20:16:59.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.373+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab900adc80 con 0x7fab7c072c50 2023-12-11T20:16:59.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0cfeffd640 1 -- 172.21.15.86:0/2163550721 <== osd.4 v2:172.21.15.133:6818/1433273204 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0d08138cb0 con 0x7f0cec073580 2023-12-11T20:16:59.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0cfeffd640 10 client.4274.objecter ms_dispatch 0x7f0d08000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:16:59.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0cfeffd640 10 client.4274.objecter _finish_command 2 = system:0 2023-12-11T20:16:59.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0cfeffd640 20 client.4274.objecter put_session s=0x7f0cec072eb0 osd=4 4 2023-12-11T20:16:59.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0cfeffd640 15 client.4274.objecter _session_command_op_remove 4 2 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 10 librados: watch_flush enter 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 10 librados: watch_flush exit 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 10 client.4274.objecter close_session for osd.4 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f0cec073580 msgr2=0x7f0cec075a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f0cec073580 0x7f0cec075a20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 20 client.4274.objecter put_session s=0x7f0cec072eb0 osd=4 1 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 20 client.4274.objecter shutdown clearing up homeless session... 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 10 client.4274.objecter successfully canceled tick 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0cec06c880 msgr2=0x7f0cec06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0cec06c880 0x7f0cec06ed40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8035000 tx=0x7f0cf8030000 comp rx=0 tx=0).stop 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 10 monclient: shutdown 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d080a8730 msgr2=0x7f0d08131660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0f175640 1 --2- 172.21.15.86:0/2163550721 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d080a8730 0x7f0d08131660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400b4a0 tx=0x7f0d0400b960 comp rx=0 tx=0).stop 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.374+0000 7f0d0e974640 1 -- 172.21.15.86:0/2163550721 reap_dead start 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.375+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 shutdown_connections 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.375+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 >> 172.21.15.86:0/2163550721 conn(0x7f0d08061900 msgr2=0x7f0d080b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.375+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 shutdown_connections 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.375+0000 7f0d0f175640 1 -- 172.21.15.86:0/2163550721 wait complete. 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.375+0000 7f0d0f175640 1 librados: shutdown 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.380+0000 7f4aa67fc640 1 -- 172.21.15.86:0/684777781 <== osd.0 v2:172.21.15.86:6820/1914311689 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4a8c001240 con 0x7f4a94072c00 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.380+0000 7f4aa67fc640 10 client.4293.objecter ms_dispatch 0x7f4ab0000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:16:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.380+0000 7f4aa67fc640 10 client.4293.objecter _finish_command 2 = system:0 2023-12-11T20:16:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.380+0000 7f4aa67fc640 20 client.4293.objecter put_session s=0x7f4a94072530 osd=0 4 2023-12-11T20:16:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.380+0000 7f4aa67fc640 15 client.4293.objecter _session_command_op_remove 0 2 2023-12-11T20:16:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 10 librados: watch_flush enter 2023-12-11T20:16:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 10 librados: watch_flush exit 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 10 client.4293.objecter close_session for osd.0 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f4a94072c00 msgr2=0x7f4a94075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f4a94072c00 0x7f4a94075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 20 client.4293.objecter put_session s=0x7f4a94072530 osd=0 1 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 20 client.4293.objecter shutdown clearing up homeless session... 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 10 client.4293.objecter successfully canceled tick 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4a9406c820 msgr2=0x7f4a9406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4a9406c820 0x7f4a9406ece0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0015d0 tx=0x7f4aac009040 comp rx=0 tx=0).stop 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 10 monclient: shutdown 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80c1990 msgr2=0x7f4aa8160240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 --2- 172.21.15.86:0/684777781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80c1990 0x7f4aa8160240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0068520 tx=0x7f4ab0067960 comp rx=0 tx=0).stop 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4ab6682640 1 -- 172.21.15.86:0/684777781 reap_dead start 2023-12-11T20:16:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 shutdown_connections 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 >> 172.21.15.86:0/684777781 conn(0x7f4aa8006610 msgr2=0x7f4aa80c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 shutdown_connections 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 -- 172.21.15.86:0/684777781 wait complete. 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.381+0000 7f4a93fff640 1 librados: shutdown 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.384+0000 7fab8cff9640 1 -- 172.21.15.86:0/2485929340 <== osd.6 v2:172.21.15.133:6826/1611390427 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fab800096c0 con 0x7fab7c072c50 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.384+0000 7fab8cff9640 10 client.4277.objecter ms_dispatch 0x7fab90000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.384+0000 7fab8cff9640 10 client.4277.objecter _finish_command 2 = system:0 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.384+0000 7fab8cff9640 20 client.4277.objecter put_session s=0x7fab7c072580 osd=6 4 2023-12-11T20:16:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.384+0000 7fab8cff9640 15 client.4277.objecter _session_command_op_remove 6 2 2023-12-11T20:16:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 10 librados: watch_flush enter 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 10 librados: watch_flush exit 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 10 client.4277.objecter close_session for osd.6 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7fab7c072c50 msgr2=0x7fab7c075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7fab7c072c50 0x7fab7c075070 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 20 client.4277.objecter put_session s=0x7fab7c072580 osd=6 1 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 20 client.4277.objecter shutdown clearing up homeless session... 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 10 client.4277.objecter successfully canceled tick 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fab7c06c870 msgr2=0x7fab7c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fab7c06c870 0x7fab7c06ed30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab8403b000 tx=0x7fab84008040 comp rx=0 tx=0).stop 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 10 monclient: shutdown 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab9006f800 msgr2=0x7fab901debd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 --2- 172.21.15.86:0/2485929340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab9006f800 0x7fab901debd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab8800ba50 tx=0x7fab8800bf10 comp rx=0 tx=0).stop 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab94f10640 1 -- 172.21.15.86:0/2485929340 reap_dead start 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 shutdown_connections 2023-12-11T20:16:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 >> 172.21.15.86:0/2485929340 conn(0x7fab900b3b20 msgr2=0x7fab9006ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 shutdown_connections 2023-12-11T20:16:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 -- 172.21.15.86:0/2485929340 wait complete. 2023-12-11T20:16:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.385+0000 7fab96973640 1 librados: shutdown 2023-12-11T20:16:59.429 INFO:teuthology.orchestra.run.smithi086.stdout:60129542148 2023-12-11T20:16:59.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-11T20:16:59.430 INFO:teuthology.orchestra.run.smithi086.stdout:68719476743 2023-12-11T20:16:59.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-11T20:16:59.431 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:16:59.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-11T20:16:59.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7f0535724640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7f0535724640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7f0535724640 10 monclient: monmap: 2023-12-11T20:16:59.608 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.610560+0000 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.610560+0000 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.609 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7fa6e6580640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7fa6e6580640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7fa6e6580640 10 monclient: monmap: 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.611136+0000 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.611136+0000 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.610 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.611 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.611 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.611 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.611 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.611 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7f4e7c1f0640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7f4e7c1f0640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7f4e7c1f0640 10 monclient: monmap: 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.611228+0000 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.611228+0000 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.613 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.613 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.613 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.610+0000 7f0535724640 1 Processor -- start 2023-12-11T20:16:59.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.611+0000 7f0535724640 1 -- start start 2023-12-11T20:16:59.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.611+0000 7f0535724640 10 monclient: init 2023-12-11T20:16:59.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.611+0000 7fa6e6580640 1 Processor -- start 2023-12-11T20:16:59.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.611+0000 7f0535724640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.611+0000 7f0535724640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-12-11T20:16:59.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05300a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:59.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f05301461c0 0x7f05300a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 10 monclient(hunting): picked mon.noname-a con 0x7f05301461c0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f053013f0f0 0x7f053013f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 10 monclient(hunting): picked mon.noname-c con 0x7f053013f0f0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0530145620 con 0x7f05300a8a20 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300aace0 con 0x7f05301461c0 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300aafc0 con 0x7f053013f0f0 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f0535724640 10 monclient(hunting): authenticate will time out at 1505.941281s 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.611+0000 7fa6e6580640 1 -- start start 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.611+0000 7fa6e6580640 10 monclient: init 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fa6e006c3e0 0x7fa6e006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient(hunting): picked mon.noname-c con 0x7fa6e006c3e0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6e0070e70 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e0143340 0x7fa6e0143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient(hunting): picked mon.noname-b con 0x7fa6e0143340 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e0070770 con 0x7fa6e0070e70 2023-12-11T20:16:59.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e00719c0 con 0x7fa6e006c3e0 2023-12-11T20:16:59.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e006ce60 con 0x7fa6e0143340 2023-12-11T20:16:59.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7fa6e6580640 10 monclient(hunting): authenticate will time out at 1505.941961s 2023-12-11T20:16:59.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.612+0000 7f4e7c1f0640 1 Processor -- start 2023-12-11T20:16:59.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f052f7fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f053013f0f0 0x7f053013f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f052f7fe640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f053013f0f0 0x7f053013f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52298/0 (socket says 172.21.15.86:52298) 2023-12-11T20:16:59.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f052f7fe640 1 -- 172.21.15.86:0/1140127957 learned_addr learned my addr 172.21.15.86:0/1140127957 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f052f7fe640 10 monclient(hunting): get_auth_request con 0x7f053013f0f0 auth_method 0 2023-12-11T20:16:59.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f052f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f052f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f052f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7fa6e557e640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fa6e006c3e0 0x7fa6e006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f4e7c1f0640 1 -- start start 2023-12-11T20:16:59.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.613+0000 7f4e7c1f0640 10 monclient: init 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7ff824b73640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7ff824b73640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7ff824b73640 10 monclient: monmap: 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.614830+0000 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.614830+0000 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.621 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.622 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.622 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.622 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.622 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-12-11T20:16:59.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e740b0b80 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:59.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4e74143d00 0x7f4e74071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient(hunting): picked mon.noname-c con 0x7f4e74143d00 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7406f4e0 0x7f4e74071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient(hunting): picked mon.noname-b con 0x7f4e7406f4e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7406ff50 con 0x7f4e740b0b80 2023-12-11T20:16:59.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e740b0f60 con 0x7f4e74143d00 2023-12-11T20:16:59.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740703e0 con 0x7f4e7406f4e0 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7f4e7c1f0640 10 monclient(hunting): authenticate will time out at 1505.943333s 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7fa6e5d7f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e0143340 0x7fa6e0143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7fa6e4d7d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fa6e0070e70 0x7fa6e00712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:59912/0 (socket says 172.21.15.86:59912) 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7fa6e4d7d640 1 -- 172.21.15.86:0/935388750 learned_addr learned my addr 172.21.15.86:0/935388750 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.614+0000 7fa6e557e640 10 monclient(hunting): get_auth_request con 0x7fa6e006c3e0 auth_method 0 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7fa6e557e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7fa6e557e640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7fa6e557e640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7fa6e5d7f640 10 monclient(hunting): get_auth_request con 0x7fa6e0143340 auth_method 0 2023-12-11T20:16:59.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7fa6e5d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7fa6e5d7f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7fa6e5d7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7ff824b73640 1 Processor -- start 2023-12-11T20:16:59.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f052e7fc640 1 --2- 172.21.15.86:0/1140127957 >> v2:172.21.15.86:3300/0 conn(0x7f05301461c0 0x7f05300a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f4e7a78d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7406f4e0 0x7f4e74071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7ff824b73640 1 -- start start 2023-12-11T20:16:59.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7ff824b73640 10 monclient: init 2023-12-11T20:16:59.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f4e7a78d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7406f4e0 0x7f4e74071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40036/0 (socket says 172.21.15.86:40036) 2023-12-11T20:16:59.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f4e7a78d640 1 -- 172.21.15.86:0/3561342866 learned_addr learned my addr 172.21.15.86:0/3561342866 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f4e7a78d640 10 monclient(hunting): get_auth_request con 0x7f4e7406f4e0 auth_method 0 2023-12-11T20:16:59.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f4e7a78d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f4e7a78d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f4e7a78d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f052f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f052f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.615+0000 7f052f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7f052e7fc640 10 monclient(hunting): get_auth_request con 0x7f05301461c0 auth_method 0 2023-12-11T20:16:59.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7f052e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7f052e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7f052e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8200af1a0 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff82006f970 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7ff820072580 0x7ff820070550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 10 monclient(hunting): picked mon.noname-c con 0x7ff820072580 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82006f4e0 con 0x7ff8200af1a0 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff820071ac0 con 0x7ff82006f970 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8200af580 con 0x7ff820072580 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7ff824b73640 10 monclient(hunting): authenticate will time out at 1505.945515s 2023-12-11T20:16:59.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7fa6e557e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7fa6e557e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7fa6e557e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7fa6e5d7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7fa6e5d7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.616+0000 7fa6e5d7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.617+0000 7ff81e59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff8200af1a0 0x7ff820071e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45300/0 (socket says 172.21.15.86:45300) 2023-12-11T20:16:59.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.617+0000 7ff81e59c640 1 -- 172.21.15.86:0/2243553145 learned_addr learned my addr 172.21.15.86:0/2243553145 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:16:59.633 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.617+0000 7f4e7a78d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.633 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.617+0000 7f4e7a78d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.633 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.617+0000 7f4e7a78d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.617+0000 7f738c6bb640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.617+0000 7f738c6bb640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.617+0000 7f738c6bb640 10 monclient: monmap: 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.618394+0000 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.618394+0000 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.618+0000 7f738c6bb640 1 Processor -- start 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.618+0000 7f738c6bb640 1 -- start start 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.618+0000 7f738c6bb640 10 monclient: init 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f052e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f052e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f052e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f73840b0b80 0x7f7384141f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient(hunting): picked mon.noname-c con 0x7f73840b0b80 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73840adc10 addr v1:172.21.15.86:6789/0 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f738413f5c0 0x7f7384142480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient(hunting): picked mon.noname-a con 0x7f738413f5c0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7384072ab0 con 0x7f73840adc10 2023-12-11T20:16:59.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738406f800 con 0x7f738413f5c0 2023-12-11T20:16:59.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73840b0f60 con 0x7f73840b0b80 2023-12-11T20:16:59.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.619+0000 7f738c6bb640 10 monclient(hunting): authenticate will time out at 1505.948693s 2023-12-11T20:16:59.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2023-12-11T20:16:59.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 1 -- 172.21.15.86:0/935388750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e0143340 msgr2=0x7fa6e0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 1 --2- 172.21.15.86:0/935388750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e0143340 0x7fa6e0143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 1 -- 172.21.15.86:0/935388750 >> v1:172.21.15.133:6789/0 conn(0x7fa6e0070e70 legacy=0x7fa6e00712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7fa6e557e640 1 -- 172.21.15.86:0/935388750 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e006b900 con 0x7fa6e006c3e0 2023-12-11T20:16:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7f967bcf2640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7f967bcf2640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.620+0000 7f967bcf2640 10 monclient: monmap: 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.621374+0000 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.621374+0000 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.639 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.640 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.640 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.621+0000 7f4e7978b640 1 --2- 172.21.15.86:0/3561342866 >> v2:172.21.15.133:3300/0 conn(0x7f4e74143d00 0x7f4e74071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.621+0000 7f967bcf2640 1 Processor -- start 2023-12-11T20:16:59.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.621+0000 7f967bcf2640 1 -- start start 2023-12-11T20:16:59.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.621+0000 7f74b1a90640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.621+0000 7f74b1a90640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f74b1a90640 10 monclient: monmap: 2023-12-11T20:16:59.641 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.641 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.641 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.622514+0000 2023-12-11T20:16:59.641 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.622514+0000 2023-12-11T20:16:59.641 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.642 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.642 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.642 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.642 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.642 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.642 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.642 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 10 monclient: init 2023-12-11T20:16:59.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-12-11T20:16:59.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f74b1a90640 1 Processor -- start 2023-12-11T20:16:59.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96740a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:59.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f96741461c0 0x7f96740a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f74b1a90640 1 -- start start 2023-12-11T20:16:59.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f74b1a90640 10 monclient: init 2023-12-11T20:16:59.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 10 monclient(hunting): picked mon.noname-a con 0x7f96741461c0 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f967413f0f0 0x7f967413f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 10 monclient(hunting): picked mon.noname-b con 0x7f967413f0f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.622+0000 7f967bcf2640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9674145620 con 0x7f96740a8a20 2023-12-11T20:16:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f967bcf2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f967bcf2640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96740aace0 con 0x7f96741461c0 2023-12-11T20:16:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f967bcf2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f967bcf2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96740aafc0 con 0x7f967413f0f0 2023-12-11T20:16:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f967bcf2640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f967bcf2640 10 monclient(hunting): authenticate will time out at 1505.952150s 2023-12-11T20:16:59.647 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f4e7978b640 10 monclient(hunting): get_auth_request con 0x7f4e74143d00 auth_method 0 2023-12-11T20:16:59.647 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f4e7978b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.647 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f4e7978b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.647 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f4e7978b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f74ac061190 0x7f74ac13ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient(hunting): picked mon.noname-a con 0x7f74ac061190 addr v2:172.21.15.86:3300/0 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f74ac13d3b0 0x7f74ac143290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient(hunting): picked mon.noname-c con 0x7f74ac13d3b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac1437d0 0x7f74ac070b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient(hunting): picked mon.noname-b con 0x7f74ac1437d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0a8c20 con 0x7f74ac061190 2023-12-11T20:16:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0aee90 con 0x7f74ac13d3b0 2023-12-11T20:16:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac061570 con 0x7f74ac1437d0 2023-12-11T20:16:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.623+0000 7f74b1a90640 10 monclient(hunting): authenticate will time out at 1505.952874s 2023-12-11T20:16:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2023-12-11T20:16:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 1 -- 172.21.15.86:0/3561342866 >> v2:172.21.15.133:3300/0 conn(0x7f4e74143d00 msgr2=0x7f4e74071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 1 --2- 172.21.15.86:0/3561342866 >> v2:172.21.15.133:3300/0 conn(0x7f4e74143d00 0x7f4e74071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 1 -- 172.21.15.86:0/3561342866 >> v1:172.21.15.133:6789/0 conn(0x7f4e740b0b80 legacy=0x7f4e741435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f4e7a78d640 1 -- 172.21.15.86:0/3561342866 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7406fbe0 con 0x7f4e7406f4e0 2023-12-11T20:16:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2023-12-11T20:16:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 1 -- 172.21.15.86:0/1140127957 >> v2:172.21.15.133:3300/0 conn(0x7f053013f0f0 msgr2=0x7f053013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 1 --2- 172.21.15.86:0/1140127957 >> v2:172.21.15.133:3300/0 conn(0x7f053013f0f0 0x7f053013f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 1 -- 172.21.15.86:0/1140127957 >> v1:172.21.15.86:6789/0 conn(0x7f05300a8a20 legacy=0x7f0530145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.653 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f052e7fc640 1 -- 172.21.15.86:0/1140127957 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053013fa90 con 0x7f05301461c0 2023-12-11T20:16:59.653 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f9875d05640 10 monclient: get_monmap_and_config 2023-12-11T20:16:59.653 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.624+0000 7f9875d05640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.653 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.625+0000 7f9875d05640 10 monclient: monmap: 2023-12-11T20:16:59.653 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.653 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:16:59.625497+0000 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:16:59.625497+0000 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.625+0000 7f9875d05640 1 Processor -- start 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.625+0000 7f9875d05640 1 -- start start 2023-12-11T20:16:59.654 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.625+0000 7f9875d05640 10 monclient: init 2023-12-11T20:16:59.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.625+0000 7fa6e557e640 1 --2- 172.21.15.86:0/935388750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e006c3e0 0x7fa6e006c7c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00097d0 tx=0x7fa6d002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ef86aa8f265ae0c server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f967a28f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f967413f0f0 0x7f967413f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98700a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f98701461c0 0x7f98700a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient(hunting): picked mon.noname-c con 0x7f98701461c0 addr v2:172.21.15.133:3300/0 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f987013f0f0 0x7f987013f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient(hunting): picked mon.noname-b con 0x7f987013f0f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9870145620 con 0x7f98700a8a20 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98700aace0 con 0x7f98701461c0 2023-12-11T20:16:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98700aafc0 con 0x7f987013f0f0 2023-12-11T20:16:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f9875d05640 10 monclient(hunting): authenticate will time out at 1505.955693s 2023-12-11T20:16:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.626+0000 7f74abfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac1437d0 0x7f74ac070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.627+0000 7f738ac58640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f738413f5c0 0x7f7384142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.627+0000 7f967a28f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f967413f0f0 0x7f967413f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40044/0 (socket says 172.21.15.86:40044) 2023-12-11T20:16:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.627+0000 7f967a28f640 1 -- 172.21.15.86:0/932095277 learned_addr learned my addr 172.21.15.86:0/932095277 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.627+0000 7f967a28f640 10 monclient(hunting): get_auth_request con 0x7f967413f0f0 auth_method 0 2023-12-11T20:16:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.627+0000 7f967a28f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.627+0000 7f967a28f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.627+0000 7f967a28f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.627+0000 7f74abfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac1437d0 0x7f74ac070b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40050/0 (socket says 172.21.15.86:40050) 2023-12-11T20:16:59.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f74abfff640 1 -- 172.21.15.86:0/411511509 learned_addr learned my addr 172.21.15.86:0/411511509 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f74abfff640 10 monclient(hunting): get_auth_request con 0x7f74ac1437d0 auth_method 0 2023-12-11T20:16:59.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f74abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f74abfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f74abfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.663 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f738ac58640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f738413f5c0 0x7f7384142480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34906/0 (socket says 172.21.15.86:34906) 2023-12-11T20:16:59.663 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f738ac58640 1 -- 172.21.15.86:0/275018633 learned_addr learned my addr 172.21.15.86:0/275018633 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.663 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f738ac58640 10 monclient(hunting): get_auth_request con 0x7f738413f5c0 auth_method 0 2023-12-11T20:16:59.663 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f738ac58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.663 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f738ac58640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f738ac58640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.628+0000 7f4e7a78d640 1 --2- 172.21.15.86:0/3561342866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7406f4e0 0x7f4e74071d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009e50 tx=0x7f4e6c02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a9dc06a0b11923f server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f738ac58640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f738ac58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f738ac58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f052e7fc640 1 --2- 172.21.15.86:0/1140127957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05301461c0 0x7f05300a8e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05300b3990 tx=0x7f052802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368755e70414c51 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f967a28f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f967a28f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f967a28f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.629+0000 7f74abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2023-12-11T20:16:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 1 -- 172.21.15.86:0/275018633 >> v2:172.21.15.133:3300/0 conn(0x7f73840b0b80 msgr2=0x7f7384141f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 1 --2- 172.21.15.86:0/275018633 >> v2:172.21.15.133:3300/0 conn(0x7f73840b0b80 0x7f7384141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 1 -- 172.21.15.86:0/275018633 >> v1:172.21.15.86:6789/0 conn(0x7f73840adc10 legacy=0x7f738413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.631+0000 7f738ac58640 1 -- 172.21.15.86:0/275018633 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738406fa00 con 0x7f738413f5c0 2023-12-11T20:16:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.632+0000 7ff81ed9d640 1 --2- 172.21.15.86:0/2243553145 >> v2:172.21.15.133:3300/0 conn(0x7ff820072580 0x7ff820070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.632+0000 7ff81d59a640 1 -- 172.21.15.86:0/2243553145 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687742300 0 0) 0x7ff82006f4e0 con 0x7ff8200af1a0 2023-12-11T20:16:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.632+0000 7ff81d59a640 10 monclient(hunting): _negotiate 2023-12-11T20:16:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.632+0000 7ff81d59a640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.632+0000 7ff81d59a640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.632+0000 7ff81d59a640 10 monclient(hunting): my global_id is 4302 2023-12-11T20:16:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.632+0000 7ff81d59a640 1 -- 172.21.15.86:0/2243553145 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff808003520 con 0x7ff8200af1a0 2023-12-11T20:16:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.632+0000 7f74ab7fe640 1 --2- 172.21.15.86:0/411511509 >> v2:172.21.15.86:3300/0 conn(0x7f74ac061190 0x7f74ac13ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f967928d640 1 --2- 172.21.15.86:0/932095277 >> v2:172.21.15.86:3300/0 conn(0x7f96741461c0 0x7f96740a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2023-12-11T20:16:59.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 1 -- 172.21.15.86:0/411511509 >> v2:172.21.15.133:3300/0 conn(0x7f74ac13d3b0 msgr2=0x7f74ac143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 1 --2- 172.21.15.86:0/411511509 >> v2:172.21.15.133:3300/0 conn(0x7f74ac13d3b0 0x7f74ac143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 1 -- 172.21.15.86:0/411511509 >> v2:172.21.15.86:3300/0 conn(0x7f74ac061190 msgr2=0x7f74ac13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 1 --2- 172.21.15.86:0/411511509 >> v2:172.21.15.86:3300/0 conn(0x7f74ac061190 0x7f74ac13ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.633+0000 7f74abfff640 1 -- 172.21.15.86:0/411511509 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac143c40 con 0x7f74ac1437d0 2023-12-11T20:16:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2023-12-11T20:16:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 1 -- 172.21.15.86:0/932095277 >> v2:172.21.15.86:3300/0 conn(0x7f96741461c0 msgr2=0x7f96740a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 1 --2- 172.21.15.86:0/932095277 >> v2:172.21.15.86:3300/0 conn(0x7f96741461c0 0x7f96740a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 1 -- 172.21.15.86:0/932095277 >> v1:172.21.15.133:6789/0 conn(0x7f96740a8a20 legacy=0x7f9674145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7f967a28f640 1 -- 172.21.15.86:0/932095277 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967413fa90 con 0x7f967413f0f0 2023-12-11T20:16:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7ff81ed9d640 10 monclient(hunting): get_auth_request con 0x7ff820072580 auth_method 0 2023-12-11T20:16:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7ff81ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7ff81ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.635+0000 7ff81ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.639+0000 7f74abfff640 1 --2- 172.21.15.86:0/411511509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac1437d0 0x7f74ac070b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f74ac072350 tx=0x7f74a402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17da140bfd4c4996 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 1 -- 172.21.15.86:0/3561342866 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c02ca80 con 0x7f4e7406f4e0 2023-12-11T20:16:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: dump: 2023-12-11T20:16:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.676 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7ff81ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7ff81ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7ff81ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e7c1f0640 5 monclient: authenticate success, global_id 4283 2023-12-11T20:16:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e7c1f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 1 -- 172.21.15.86:0/3561342866 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c02cbe0 con 0x7f4e7406f4e0 2023-12-11T20:16:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e7c1f0640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e7c1f0640 4 set_mon_vals no callback set 2023-12-11T20:16:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 1 -- 172.21.15.86:0/3561342866 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0343f0 con 0x7f4e7406f4e0 2023-12-11T20:16:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.640+0000 7f4e78f8a640 10 monclient: dump: 2023-12-11T20:16:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 10 monclient: shutdown 2023-12-11T20:16:59.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f967a28f640 1 --2- 172.21.15.86:0/932095277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f967413f0f0 0x7f967413f550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f966c009e50 tx=0x7f966c02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32c97316c420c10b server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/3561342866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7406f4e0 msgr2=0x7f4e74071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/3561342866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7406f4e0 0x7f4e74071d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009e50 tx=0x7f4e6c02bee0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/3561342866 shutdown_connections 2023-12-11T20:16:59.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/3561342866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7406f4e0 0x7f4e74071d70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/3561342866 >> v2:172.21.15.133:3300/0 conn(0x7f4e74143d00 0x7f4e74071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/3561342866 >> 172.21.15.86:0/3561342866 conn(0x7f4e740b3ef0 msgr2=0x7f4e740b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/3561342866 shutdown_connections 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.641+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/3561342866 wait complete. 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 10 monclient: monmap: 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.683 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7ff81ed9d640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2023-12-11T20:16:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7ff81ed9d640 1 -- 172.21.15.86:0/2243553145 >> v1:172.21.15.133:6789/0 conn(0x7ff82006f970 legacy=0x7ff82013b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7ff81ed9d640 1 -- 172.21.15.86:0/2243553145 >> v1:172.21.15.86:6789/0 conn(0x7ff8200af1a0 legacy=0x7ff820071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7ff81ed9d640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7ff81ed9d640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7ff81ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7ff81ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7ff81ed9d640 1 -- 172.21.15.86:0/2243553145 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8200adaa0 con 0x7ff820072580 2023-12-11T20:16:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 1 librados: starting msgr at 2023-12-11T20:16:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 1 librados: starting objecter 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 1 Processor -- start 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 1 -- start start 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 1 librados: setting wanted keys 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 1 librados: calling monclient init 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.642+0000 7f4e7c1f0640 10 monclient: init 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7406f4e0 0x7f4e740a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient(hunting): picked mon.noname-a con 0x7f4e7406f4e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4e740b0b80 0x7f4e740a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient(hunting): picked mon.noname-b con 0x7f4e740b0b80 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e74143d00 0x7f4e741443c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient(hunting): picked mon.noname-c con 0x7f4e74143d00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740a8a00 con 0x7f4e7406f4e0 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740a9b20 con 0x7f4e740b0b80 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e74146e60 con 0x7f4e74143d00 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.643+0000 7f4e7c1f0640 10 monclient(hunting): authenticate will time out at 1505.972781s 2023-12-11T20:16:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.644+0000 7f738ac58640 1 --2- 172.21.15.86:0/275018633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f738413f5c0 0x7f7384142480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73840ad210 tx=0x7f737c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c31460d1d2f812 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7f986effd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f98701461c0 0x7f98700a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 1 -- 172.21.15.86:0/935388750 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0031780 con 0x7fa6e006c3e0 2023-12-11T20:16:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: dump: 2023-12-11T20:16:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 1 -- 172.21.15.86:0/935388750 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d00318e0 con 0x7fa6e006c3e0 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6e6580640 5 monclient: authenticate success, global_id 4360 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6e6580640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6e6580640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6e6580640 4 set_mon_vals no callback set 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 1 -- 172.21.15.86:0/935388750 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0031c70 con 0x7fa6e006c3e0 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.645+0000 7fa6d7fff640 10 monclient: dump: 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.691 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 1 -- 172.21.15.86:0/411511509 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a402c9e0 con 0x7f74ac1437d0 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: dump: 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 1 -- 172.21.15.86:0/411511509 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a402cb40 con 0x7f74ac1437d0 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74b1a90640 5 monclient: authenticate success, global_id 4289 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74b1a90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74b1a90640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74b1a90640 4 set_mon_vals no callback set 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 1 -- 172.21.15.86:0/411511509 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a40343f0 con 0x7f74ac1437d0 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74aa7fc640 10 monclient: dump: 2023-12-11T20:16:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f986effd640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f98701461c0 0x7f98700a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52354/0 (socket says 172.21.15.86:52354) 2023-12-11T20:16:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f986effd640 1 -- 172.21.15.86:0/1360034651 learned_addr learned my addr 172.21.15.86:0/1360034651 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f986effd640 10 monclient(hunting): get_auth_request con 0x7f98701461c0 auth_method 0 2023-12-11T20:16:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f986effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f986effd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f986effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 10 monclient: shutdown 2023-12-11T20:16:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 1 -- 172.21.15.86:0/935388750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e006c3e0 msgr2=0x7fa6e006c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 1 --2- 172.21.15.86:0/935388750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e006c3e0 0x7fa6e006c7c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00097d0 tx=0x7fa6d002b820 comp rx=0 tx=0).stop 2023-12-11T20:16:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 1 -- 172.21.15.86:0/935388750 shutdown_connections 2023-12-11T20:16:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 1 --2- 172.21.15.86:0/935388750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e0143340 0x7fa6e0143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 1 --2- 172.21.15.86:0/935388750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e006c3e0 0x7fa6e006c7c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 1 -- 172.21.15.86:0/935388750 >> 172.21.15.86:0/935388750 conn(0x7fa6e00b5690 msgr2=0x7fa6e00b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 1 -- 172.21.15.86:0/1140127957 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052802cc80 con 0x7f05301461c0 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: dump: 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 1 -- 172.21.15.86:0/1140127957 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052802cde0 con 0x7f05301461c0 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 1 -- 172.21.15.86:0/1140127957 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528034660 con 0x7f05301461c0 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f052dffb640 10 monclient: dump: 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f0535724640 5 monclient: authenticate success, global_id 4299 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f0535724640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f0535724640 4 set_mon_vals no callback set 2023-12-11T20:16:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f4e79f8c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7406f4e0 0x7f4e740a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.700 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 1 -- 172.21.15.86:0/935388750 shutdown_connections 2023-12-11T20:16:59.700 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74b1a90640 10 monclient: shutdown 2023-12-11T20:16:59.700 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74b1a90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.700 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74b1a90640 1 -- 172.21.15.86:0/411511509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac1437d0 msgr2=0x7f74ac070b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.700 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f74b1a90640 1 --2- 172.21.15.86:0/411511509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac1437d0 0x7f74ac070b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f74ac072350 tx=0x7f74a402be40 comp rx=0 tx=0).stop 2023-12-11T20:16:59.700 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7f986ffff640 1 --2- 172.21.15.86:0/1360034651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f987013f0f0 0x7f987013f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.646+0000 7fa6e6580640 1 -- 172.21.15.86:0/935388750 wait complete. 2023-12-11T20:16:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f74b1a90640 1 -- 172.21.15.86:0/411511509 shutdown_connections 2023-12-11T20:16:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f74b1a90640 1 --2- 172.21.15.86:0/411511509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac1437d0 0x7f74ac070b60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f74b1a90640 1 --2- 172.21.15.86:0/411511509 >> v2:172.21.15.133:3300/0 conn(0x7f74ac13d3b0 0x7f74ac143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f74b1a90640 1 --2- 172.21.15.86:0/411511509 >> v2:172.21.15.86:3300/0 conn(0x7f74ac061190 0x7f74ac13ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f74b1a90640 1 -- 172.21.15.86:0/411511509 >> 172.21.15.86:0/411511509 conn(0x7f74ac0b3e50 msgr2=0x7f74ac0b4260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7406f4e0 0x7f4e740a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34922/0 (socket says 172.21.15.86:34922) 2023-12-11T20:16:59.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f986effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f986effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f986effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 1 -- 172.21.15.86:0/1284314290 learned_addr learned my addr 172.21.15.86:0/1284314290 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 10 monclient(hunting): get_auth_request con 0x7f4e7406f4e0 auth_method 0 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f74b1a90640 1 -- 172.21.15.86:0/411511509 shutdown_connections 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f74b1a90640 1 -- 172.21.15.86:0/411511509 wait complete. 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f986ffff640 10 monclient(hunting): get_auth_request con 0x7f987013f0f0 auth_method 0 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f986ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f986ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.704 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f986ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.704 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.704 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.704 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f4e79f8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f0535724640 10 monclient: shutdown 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f0535724640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f0535724640 1 -- 172.21.15.86:0/1140127957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05301461c0 msgr2=0x7f05300a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f0535724640 1 --2- 172.21.15.86:0/1140127957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05301461c0 0x7f05300a8e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05300b3990 tx=0x7f052802bed0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7fa6e6580640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7fa6e6580640 10 monclient: monmap: 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.705 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.706 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.706 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.706 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.706 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.706 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f0535724640 1 -- 172.21.15.86:0/1140127957 shutdown_connections 2023-12-11T20:16:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f0535724640 1 --2- 172.21.15.86:0/1140127957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05301461c0 0x7f05300a8e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f0535724640 1 --2- 172.21.15.86:0/1140127957 >> v2:172.21.15.133:3300/0 conn(0x7f053013f0f0 0x7f053013f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.647+0000 7f0535724640 1 -- 172.21.15.86:0/1140127957 >> 172.21.15.86:0/1140127957 conn(0x7f05300b7890 msgr2=0x7f05300b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f0535724640 1 -- 172.21.15.86:0/1140127957 shutdown_connections 2023-12-11T20:16:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7fa6e6580640 1 librados: starting msgr at 2023-12-11T20:16:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f0535724640 1 -- 172.21.15.86:0/1140127957 wait complete. 2023-12-11T20:16:59.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7fa6e6580640 1 librados: starting objecter 2023-12-11T20:16:59.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7fa6e6580640 1 Processor -- start 2023-12-11T20:16:59.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f74b1a90640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7fa6e6580640 1 -- start start 2023-12-11T20:16:59.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7fa6e6580640 1 librados: setting wanted keys 2023-12-11T20:16:59.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7fa6e6580640 1 librados: calling monclient init 2023-12-11T20:16:59.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7fa6e6580640 10 monclient: init 2023-12-11T20:16:59.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f74b1a90640 10 monclient: monmap: 2023-12-11T20:16:59.709 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f74b1a90640 1 librados: starting msgr at 2023-12-11T20:16:59.710 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f74b1a90640 1 librados: starting objecter 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f0535724640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f0535724640 10 monclient: monmap: 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.711 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f74b1a90640 1 Processor -- start 2023-12-11T20:16:59.712 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f0535724640 1 librados: starting msgr at 2023-12-11T20:16:59.712 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 -- start start 2023-12-11T20:16:59.712 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 librados: setting wanted keys 2023-12-11T20:16:59.712 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 librados: calling monclient init 2023-12-11T20:16:59.712 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient: init 2023-12-11T20:16:59.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:16:59.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e006c3e0 0x7fa6e01441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient(hunting): picked mon.noname-c con 0x7fa6e006c3e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e0070e70 0x7fa6e01def10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient(hunting): picked mon.noname-a con 0x7fa6e0070e70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.648+0000 7f0535724640 1 librados: starting objecter 2023-12-11T20:16:59.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e0143340 0x7fa6e01df450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f4e7a78d640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e74143d00 0x7f4e741443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f0535724640 1 Processor -- start 2023-12-11T20:16:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient(hunting): picked mon.noname-b con 0x7fa6e0143340 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e014a160 con 0x7fa6e0070e70 2023-12-11T20:16:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e014a400 con 0x7fa6e0143340 2023-12-11T20:16:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e0145400 con 0x7fa6e006c3e0 2023-12-11T20:16:59.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7fa6e6580640 10 monclient(hunting): authenticate will time out at 1505.978444s 2023-12-11T20:16:59.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f0535724640 1 -- start start 2023-12-11T20:16:59.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f0535724640 1 librados: setting wanted keys 2023-12-11T20:16:59.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f0535724640 1 librados: calling monclient init 2023-12-11T20:16:59.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f0535724640 10 monclient: init 2023-12-11T20:16:59.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f4e7a78d640 10 monclient(hunting): get_auth_request con 0x7f4e74143d00 auth_method 0 2023-12-11T20:16:59.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f4e7a78d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f4e7a78d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f4e7a78d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac061190 0x7f74ac1d82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient(hunting): picked mon.noname-c con 0x7f74ac061190 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ac13d3b0 0x7f74ac1d8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient(hunting): picked mon.noname-a con 0x7f74ac13d3b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74ac1437d0 0x7f74ac138df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.718 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient(hunting): picked mon.noname-b con 0x7f74ac1437d0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac1d8e70 con 0x7f74ac13d3b0 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0ad0a0 con 0x7f74ac1437d0 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac139a90 con 0x7f74ac061190 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.649+0000 7f74b1a90640 10 monclient(hunting): authenticate will time out at 1505.978889s 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:16:59.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2023-12-11T20:16:59.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 1 -- 172.21.15.86:0/1360034651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f987013f0f0 msgr2=0x7f987013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:16:59.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 1 --2- 172.21.15.86:0/1360034651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f987013f0f0 0x7f987013f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 1 -- 172.21.15.86:0/1360034651 >> v1:172.21.15.133:6789/0 conn(0x7f98700a8a20 legacy=0x7f9870145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f986effd640 1 -- 172.21.15.86:0/1360034651 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987013fa90 con 0x7f98701461c0 2023-12-11T20:16:59.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05300a8a20 0x7f05301d7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient(hunting): picked mon.noname-b con 0x7f05300a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f053013f0f0 0x7f0530138230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient(hunting): picked mon.noname-c con 0x7f053013f0f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05301461c0 0x7f0530144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient(hunting): picked mon.noname-a con 0x7f05301461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530144720 con 0x7f05301461c0 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05301442d0 con 0x7f05300a8a20 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0530144490 con 0x7f053013f0f0 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.650+0000 7f0535724640 10 monclient(hunting): authenticate will time out at 1505.979479s 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 1 -- 172.21.15.86:0/1284314290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e74143d00 msgr2=0x7f4e741443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e74143d00 0x7f4e741443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 1 -- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4e740b0b80 msgr2=0x7f4e740a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4e740b0b80 0x7f4e740a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.651+0000 7f4e79f8c640 1 -- 172.21.15.86:0/1284314290 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c009b30 con 0x7f4e7406f4e0 2023-12-11T20:16:59.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.657+0000 7f4e79f8c640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7406f4e0 0x7f4e740a84c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400c450 tx=0x7f4e6400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.657+0000 7f4e6a7fc640 10 client.?.objecter ms_handle_connect 0x7f4e7406f4e0 2023-12-11T20:16:59.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.657+0000 7f4e6a7fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.657+0000 7ff81ed9d640 1 --2- 172.21.15.86:0/2243553145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff820070550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff818009dc0 tx=0x7ff81802be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17c788e7fabca51b server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.658+0000 7f986effd640 1 --2- 172.21.15.86:0/1360034651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98701461c0 0x7f98700a8e00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9864009870 tx=0x7f986402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58f346afb37fe0b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7fa6e557e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e006c3e0 0x7fa6e01441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 1 -- 172.21.15.86:0/275018633 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c02c960 con 0x7f738413f5c0 2023-12-11T20:16:59.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:59.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: dump: 2023-12-11T20:16:59.725 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.725 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.726 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.726 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.726 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.726 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.726 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.726 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f738c6bb640 5 monclient: authenticate success, global_id 4305 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f738c6bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 1 -- 172.21.15.86:0/275018633 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c02cac0 con 0x7f738413f5c0 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f738c6bb640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f738c6bb640 4 set_mon_vals no callback set 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f052e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f053013f0f0 0x7f0530138230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 1 -- 172.21.15.86:0/275018633 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c02ce50 con 0x7f738413f5c0 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.659+0000 7f7389455640 10 monclient: dump: 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.728 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e006c3e0 0x7fa6e01441e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40090/0 (socket says 172.21.15.86:40090) 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 1 -- 172.21.15.86:0/1520532875 learned_addr learned my addr 172.21.15.86:0/1520532875 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 10 monclient(hunting): get_auth_request con 0x7fa6e006c3e0 auth_method 0 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 10 monclient: shutdown 2023-12-11T20:16:59.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 1 -- 172.21.15.86:0/275018633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f738413f5c0 msgr2=0x7f7384142480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 1 --2- 172.21.15.86:0/275018633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f738413f5c0 0x7f7384142480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73840ad210 tx=0x7f737c02bdc0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 1 -- 172.21.15.86:0/275018633 shutdown_connections 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 1 --2- 172.21.15.86:0/275018633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f738413f5c0 0x7f7384142480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 1 --2- 172.21.15.86:0/275018633 >> v2:172.21.15.133:3300/0 conn(0x7f73840b0b80 0x7f7384141f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 1 -- 172.21.15.86:0/275018633 >> 172.21.15.86:0/275018633 conn(0x7f73840b3ef0 msgr2=0x7f73840b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 1 -- 172.21.15.86:0/275018633 shutdown_connections 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f052e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f053013f0f0 0x7f0530138230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40102/0 (socket says 172.21.15.86:40102) 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f052e7fc640 1 -- 172.21.15.86:0/3252414197 learned_addr learned my addr 172.21.15.86:0/3252414197 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f052e7fc640 10 monclient(hunting): get_auth_request con 0x7f053013f0f0 auth_method 0 2023-12-11T20:16:59.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f052e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f052e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f052e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7f738c6bb640 1 -- 172.21.15.86:0/275018633 wait complete. 2023-12-11T20:16:59.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.660+0000 7fa6e557e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 10 monclient: monmap: 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.733 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 1 -- 172.21.15.86:0/932095277 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c02ca80 con 0x7f967413f0f0 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: dump: 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 1 -- 172.21.15.86:0/932095277 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c02cbe0 con 0x7f967413f0f0 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 1 -- 172.21.15.86:0/932095277 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c0343f0 con 0x7f967413f0f0 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f9678a8c640 10 monclient: dump: 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f967bcf2640 5 monclient: authenticate success, global_id 4286 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f967bcf2640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f967bcf2640 4 set_mon_vals no callback set 2023-12-11T20:16:59.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 1 librados: starting msgr at 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 1 librados: starting objecter 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 1 Processor -- start 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 1 -- start start 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 1 librados: setting wanted keys 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 1 librados: calling monclient init 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.661+0000 7f738c6bb640 10 monclient: init 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f052e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f052e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f052e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 10 monclient: shutdown 2023-12-11T20:16:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 1 -- 172.21.15.86:0/932095277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f967413f0f0 msgr2=0x7f967413f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 1 --2- 172.21.15.86:0/932095277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f967413f0f0 0x7f967413f550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f966c009e50 tx=0x7f966c02bee0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac061190 0x7f74ac1d82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 1 -- 172.21.15.86:0/932095277 shutdown_connections 2023-12-11T20:16:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 1 --2- 172.21.15.86:0/932095277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f967413f0f0 0x7f967413f550 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 1 --2- 172.21.15.86:0/932095277 >> v2:172.21.15.86:3300/0 conn(0x7f96741461c0 0x7f96740a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 1 -- 172.21.15.86:0/932095277 >> 172.21.15.86:0/932095277 conn(0x7f96740b7890 msgr2=0x7f96740b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:16:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 1 -- 172.21.15.86:0/932095277 shutdown_connections 2023-12-11T20:16:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73840b0b80 0x7f73840af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient(hunting): picked mon.noname-a con 0x7f73840b0b80 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f738413f5c0 0x7f73840aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient(hunting): picked mon.noname-b con 0x7f738413f5c0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738413acd0 0x7f73841da3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient(hunting): picked mon.noname-c con 0x7f738413acd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f967bcf2640 1 -- 172.21.15.86:0/932095277 wait complete. 2023-12-11T20:16:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7384137840 con 0x7f73840b0b80 2023-12-11T20:16:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738413a840 con 0x7f738413f5c0 2023-12-11T20:16:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738413a510 con 0x7f738413acd0 2023-12-11T20:16:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f738c6bb640 10 monclient(hunting): authenticate will time out at 1505.991654s 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac061190 0x7f74ac1d82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40094/0 (socket says 172.21.15.86:40094) 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f74ab7fe640 1 -- 172.21.15.86:0/3937837914 learned_addr learned my addr 172.21.15.86:0/3937837914 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f74ab7fe640 10 monclient(hunting): get_auth_request con 0x7f74ac061190 auth_method 0 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f74ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f74ab7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.662+0000 7f74ab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 1 -- 172.21.15.86:0/2243553145 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81802cc40 con 0x7ff820072580 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: dump: 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 1 -- 172.21.15.86:0/2243553145 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81802cda0 con 0x7ff820072580 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 1 -- 172.21.15.86:0/2243553145 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818034660 con 0x7ff820072580 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff81d59a640 10 monclient: dump: 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.746 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.746 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.746 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff824b73640 5 monclient: authenticate success, global_id 4375 2023-12-11T20:16:59.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff824b73640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff824b73640 4 set_mon_vals no callback set 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7f967bcf2640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7f967bcf2640 10 monclient: monmap: 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.747 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7f967bcf2640 1 librados: starting msgr at 2023-12-11T20:16:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7f967bcf2640 1 librados: starting objecter 2023-12-11T20:16:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7f967bcf2640 1 Processor -- start 2023-12-11T20:16:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff824b73640 10 monclient: shutdown 2023-12-11T20:16:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff824b73640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff824b73640 1 -- 172.21.15.86:0/2243553145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 msgr2=0x7ff820070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.663+0000 7ff824b73640 1 --2- 172.21.15.86:0/2243553145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff820070550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff818009dc0 tx=0x7ff81802be90 comp rx=0 tx=0).stop 2023-12-11T20:16:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 -- start start 2023-12-11T20:16:59.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 librados: setting wanted keys 2023-12-11T20:16:59.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 librados: calling monclient init 2023-12-11T20:16:59.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient: init 2023-12-11T20:16:59.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7ff824b73640 1 -- 172.21.15.86:0/2243553145 shutdown_connections 2023-12-11T20:16:59.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7ff824b73640 1 --2- 172.21.15.86:0/2243553145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff820070550 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7ff824b73640 1 -- 172.21.15.86:0/2243553145 >> 172.21.15.86:0/2243553145 conn(0x7ff82006abe0 msgr2=0x7ff8200b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7ff824b73640 1 -- 172.21.15.86:0/2243553145 shutdown_connections 2023-12-11T20:16:59.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7ff824b73640 1 -- 172.21.15.86:0/2243553145 wait complete. 2023-12-11T20:16:59.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7fa6e4d7d640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e0070e70 0x7fa6e01def10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a8a20 0x7f9674144970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient(hunting): picked mon.noname-a con 0x7f96740a8a20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f967413f0f0 0x7f9674140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient(hunting): picked mon.noname-b con 0x7f967413f0f0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741461c0 0x7f9674140840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient(hunting): picked mon.noname-c con 0x7f96741461c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9674140d80 con 0x7f96740a8a20 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967413fe50 con 0x7f967413f0f0 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9674141870 con 0x7f96741461c0 2023-12-11T20:16:59.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.664+0000 7f967bcf2640 10 monclient(hunting): authenticate will time out at 1505.994103s 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 10 monclient: monmap: 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.752 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e4d7d640 10 monclient(hunting): get_auth_request con 0x7fa6e0070e70 auth_method 0 2023-12-11T20:16:59.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e4d7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e4d7d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e4d7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 1 librados: starting msgr at 2023-12-11T20:16:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 1 librados: starting objecter 2023-12-11T20:16:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 1 Processor -- start 2023-12-11T20:16:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 1 -- start start 2023-12-11T20:16:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 1 librados: setting wanted keys 2023-12-11T20:16:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 1 librados: calling monclient init 2023-12-11T20:16:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7ff824b73640 10 monclient: init 2023-12-11T20:16:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 1 -- 172.21.15.86:0/1284314290 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e640052d0 con 0x7f4e7406f4e0 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: dump: 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 1 -- 172.21.15.86:0/1284314290 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e64005430 con 0x7f4e7406f4e0 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 1 -- 172.21.15.86:0/1284314290 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64012870 con 0x7f4e7406f4e0 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 1 -- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e0143340 msgr2=0x7fa6e01df450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e0143340 0x7fa6e01df450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 1 -- 172.21.15.86:0/1520532875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e0070e70 msgr2=0x7fa6e01def10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e0070e70 0x7fa6e01def10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6a7fc640 10 monclient: dump: 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.757 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7fa6e557e640 1 -- 172.21.15.86:0/1520532875 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0009400 con 0x7fa6e006c3e0 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 5 monclient: authenticate success, global_id 4308 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e6bfff640 4 set_mon_vals no callback set 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 10 monclient: _renew_subs 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e740aa480 con 0x7f4e7406f4e0 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 10 monclient: _renew_subs 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7413fb30 con 0x7f4e7406f4e0 2023-12-11T20:16:59.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.665+0000 7f4e7c1f0640 1 librados: init done 2023-12-11T20:16:59.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7f74aaffd640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ac13d3b0 0x7f74ac1d8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7f052effd640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05300a8a20 0x7f05301d7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:59.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff82006f970 0x7ff82013e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 10 monclient(hunting): picked mon.noname-c con 0x7ff82006f970 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff82014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 10 monclient(hunting): picked mon.noname-b con 0x7ff820072580 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8200af1a0 0x7ff8200aca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 10 monclient(hunting): picked mon.noname-a con 0x7ff8200af1a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.666+0000 7ff824b73640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f74aaffd640 10 monclient(hunting): get_auth_request con 0x7f74ac13d3b0 auth_method 0 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f74aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f74aaffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f74aaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7ff824b73640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820135450 con 0x7ff8200af1a0 2023-12-11T20:16:59.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7ff824b73640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7ff824b73640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8201504a0 con 0x7ff820072580 2023-12-11T20:16:59.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7ff824b73640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7ff824b73640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff820150730 con 0x7ff82006f970 2023-12-11T20:16:59.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7ff824b73640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7ff824b73640 10 monclient(hunting): authenticate will time out at 1505.996222s 2023-12-11T20:16:59.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f4e7c1f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f4e7c1f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f4e7c1f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74070940 con 0x7f4e7406f4e0 2023-12-11T20:16:59.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f74aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f74aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.667+0000 7f74aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2023-12-11T20:16:59.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 1 -- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05300a8a20 msgr2=0x7f05301d7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05300a8a20 0x7f05301d7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 1 -- 172.21.15.86:0/3252414197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05301461c0 msgr2=0x7f0530144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05301461c0 0x7f0530144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 1 monclient: found mon.noname-c 2023-12-11T20:16:59.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.668+0000 7f052e7fc640 1 -- 172.21.15.86:0/3252414197 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0528009a70 con 0x7f053013f0f0 2023-12-11T20:16:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 1 -- 172.21.15.86:0/1360034651 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864030840 con 0x7f98701461c0 2023-12-11T20:16:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: dump: 2023-12-11T20:16:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 1 -- 172.21.15.86:0/1360034651 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98640309a0 con 0x7f98701461c0 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 1 -- 172.21.15.86:0/1360034651 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864030d30 con 0x7f98701461c0 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f986e7fc640 10 monclient: dump: 2023-12-11T20:16:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f9875d05640 5 monclient: authenticate success, global_id 4378 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f9875d05640 10 monclient: get_monmap_and_config success 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.669+0000 7f9875d05640 4 set_mon_vals no callback set 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 10 monclient: shutdown 2023-12-11T20:16:59.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 1 -- 172.21.15.86:0/1360034651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98701461c0 msgr2=0x7f98700a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 1 --2- 172.21.15.86:0/1360034651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98701461c0 0x7f98700a8e00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9864009870 tx=0x7f986402b980 comp rx=0 tx=0).stop 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 1 -- 172.21.15.86:0/1360034651 shutdown_connections 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 1 --2- 172.21.15.86:0/1360034651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f987013f0f0 0x7f987013f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 1 --2- 172.21.15.86:0/1360034651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98701461c0 0x7f98700a8e00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 1 -- 172.21.15.86:0/1360034651 >> 172.21.15.86:0/1360034651 conn(0x7f98700b7890 msgr2=0x7f98700b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 1 -- 172.21.15.86:0/1360034651 shutdown_connections 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f9875d05640 1 -- 172.21.15.86:0/1360034651 wait complete. 2023-12-11T20:16:59.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.670+0000 7f967928d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f967413f0f0 0x7f9674140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f9875d05640 10 monclient: build_initial_monmap 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f9875d05640 10 monclient: monmap: 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:16:59.771 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.772 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f967928d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f967413f0f0 0x7f9674140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52382/0 (socket says 172.21.15.86:52382) 2023-12-11T20:16:59.772 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f967928d640 1 -- 172.21.15.86:0/3875813906 learned_addr learned my addr 172.21.15.86:0/3875813906 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.772 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f967928d640 10 monclient(hunting): get_auth_request con 0x7f967413f0f0 auth_method 0 2023-12-11T20:16:59.772 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f967928d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.772 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f967928d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.772 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f967928d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f9875d05640 1 librados: starting msgr at 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f9875d05640 1 librados: starting objecter 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f9875d05640 1 Processor -- start 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 1 -- 172.21.15.86:0/3937837914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac061190 msgr2=0x7f74ac1d82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac061190 0x7f74ac1d82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 1 -- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74ac1437d0 msgr2=0x7f74ac138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74ac1437d0 0x7f74ac138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.671+0000 7f74aaffd640 1 -- 172.21.15.86:0/3937837914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a4009b30 con 0x7f74ac13d3b0 2023-12-11T20:16:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f9875d05640 1 -- start start 2023-12-11T20:16:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f9875d05640 1 librados: setting wanted keys 2023-12-11T20:16:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f9875d05640 1 librados: calling monclient init 2023-12-11T20:16:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f9875d05640 10 monclient: init 2023-12-11T20:16:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f967928d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f967928d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f967928d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f9875d05640 10 monclient: _reopen_session rank -1 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.672+0000 7f9875d05640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98700a8a20 0x7f98701432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 10 monclient(hunting): picked mon.noname-c con 0x7f98700a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f987013f0f0 0x7f9870143800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 10 monclient(hunting): picked mon.noname-b con 0x7f987013f0f0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98701461c0 0x7f9870137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 10 monclient(hunting): picked mon.noname-a con 0x7f98701461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98701400c0 con 0x7f98701461c0 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98701377e0 con 0x7f987013f0f0 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 10 monclient(hunting): start opening mon connection 2023-12-11T20:16:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9870143d40 con 0x7f98700a8a20 2023-12-11T20:16:59.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 10 monclient(hunting): _renew_subs 2023-12-11T20:16:59.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f9875d05640 10 monclient(hunting): authenticate will time out at 1506.002257s 2023-12-11T20:16:59.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f052e7fc640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f053013f0f0 0x7f0530138230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0528037040 tx=0x7f0528004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f05177fe640 10 client.?.objecter ms_handle_connect 0x7f053013f0f0 2023-12-11T20:16:59.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7f05177fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.778 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.673+0000 7ff81dd9b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff82014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.778 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.674+0000 7fa6e557e640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e006c3e0 0x7fa6e01441e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00097d0 tx=0x7fa6d00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.778 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.674+0000 7fa6d5ffb640 10 client.?.objecter ms_handle_connect 0x7fa6e006c3e0 2023-12-11T20:16:59.778 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.674+0000 7fa6d5ffb640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 1 -- 172.21.15.86:0/3875813906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741461c0 msgr2=0x7f9674140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741461c0 0x7f9674140840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 1 -- 172.21.15.86:0/3875813906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a8a20 msgr2=0x7f9674144970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a8a20 0x7f9674144970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.676+0000 7f967928d640 1 -- 172.21.15.86:0/3875813906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c009b30 con 0x7f967413f0f0 2023-12-11T20:16:59.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.677+0000 7f74aaffd640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ac13d3b0 0x7f74ac1d8830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f74a000b3f0 tx=0x7f74a000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.677+0000 7f7493fff640 10 client.?.objecter ms_handle_connect 0x7f74ac13d3b0 2023-12-11T20:16:59.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.677+0000 7f7493fff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.678+0000 7f967928d640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f967413f0f0 0x7f9674140300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f966400c7f0 tx=0x7f966400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.678+0000 7f966a7fc640 10 client.?.objecter ms_handle_connect 0x7f967413f0f0 2023-12-11T20:16:59.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.678+0000 7f966a7fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.679+0000 7ff81dd9b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff82014ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52392/0 (socket says 172.21.15.86:52392) 2023-12-11T20:16:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.679+0000 7ff81dd9b640 1 -- 172.21.15.86:0/1503599004 learned_addr learned my addr 172.21.15.86:0/1503599004 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.679+0000 7ff81dd9b640 10 monclient(hunting): get_auth_request con 0x7ff820072580 auth_method 0 2023-12-11T20:16:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.679+0000 7ff81dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.679+0000 7ff81dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.679+0000 7ff81dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.681+0000 7ff81ed9d640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8200af1a0 0x7ff8200aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.681+0000 7f738a457640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73840b0b80 0x7f73840af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.681+0000 7f7389c56640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f738413f5c0 0x7f73840aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7f7389c56640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f738413f5c0 0x7f73840aebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52380/0 (socket says 172.21.15.86:52380) 2023-12-11T20:16:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7f7389c56640 1 -- 172.21.15.86:0/2714356346 learned_addr learned my addr 172.21.15.86:0/2714356346 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7f7389c56640 10 monclient(hunting): get_auth_request con 0x7f738413f5c0 auth_method 0 2023-12-11T20:16:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7f7389c56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7f7389c56640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7f7389c56640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 1 -- 172.21.15.86:0/1520532875 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0032820 con 0x7fa6e006c3e0 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: dump: 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 1 -- 172.21.15.86:0/1520532875 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d00329a0 con 0x7fa6e006c3e0 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 1 -- 172.21.15.86:0/1520532875 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0032d50 con 0x7fa6e006c3e0 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d5ffb640 10 monclient: dump: 2023-12-11T20:16:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6d77fe640 4 set_mon_vals no callback set 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6e6580640 5 monclient: authenticate success, global_id 4298 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6e6580640 10 monclient: _renew_subs 2023-12-11T20:16:59.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6e6580640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.682+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e00afd80 con 0x7fa6e006c3e0 2023-12-11T20:16:59.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7fa6e6580640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7fa6e6580640 10 monclient: _renew_subs 2023-12-11T20:16:59.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7fa6e6580640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e01dfb70 con 0x7fa6e006c3e0 2023-12-11T20:16:59.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7fa6e6580640 1 librados: init done 2023-12-11T20:16:59.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 1 -- 172.21.15.86:0/3252414197 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528040650 con 0x7f053013f0f0 2023-12-11T20:16:59.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7ff81ed9d640 10 monclient(hunting): get_auth_request con 0x7ff8200af1a0 auth_method 0 2023-12-11T20:16:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7ff81ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:16:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7ff81ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7ff81ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: dump: 2023-12-11T20:16:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 1 -- 172.21.15.86:0/3252414197 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05280407b0 con 0x7f053013f0f0 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 5 monclient: authenticate success, global_id 4301 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 10 monclient: _renew_subs 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05301450f0 con 0x7f053013f0f0 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 10 monclient: _renew_subs 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0530143ff0 con 0x7f053013f0f0 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f0535724640 1 librados: init done 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f052d7fa640 4 set_mon_vals no callback set 2023-12-11T20:16:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 1 -- 172.21.15.86:0/3252414197 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528040b60 con 0x7f053013f0f0 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.683+0000 7f05177fe640 10 monclient: dump: 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.794 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f738ac58640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738413acd0 0x7f73841da3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7fa6e6580640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7fa6e6580640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7fa6e6580640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e0075a40 con 0x7fa6e006c3e0 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 1 -- 172.21.15.86:0/3875813906 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664004220 con 0x7f967413f0f0 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: dump: 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.796 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7ff81dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7ff81dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7ff81dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 1 -- 172.21.15.86:0/3875813906 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9664004380 con 0x7f967413f0f0 2023-12-11T20:16:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 5 monclient: authenticate success, global_id 4384 2023-12-11T20:16:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 10 monclient: _renew_subs 2023-12-11T20:16:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9674137190 con 0x7f967413f0f0 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 1 -- 172.21.15.86:0/3875813906 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664007840 con 0x7f967413f0f0 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966a7fc640 10 monclient: dump: 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 10 monclient: _renew_subs 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f966bfff640 4 set_mon_vals no callback set 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96741376a0 con 0x7f967413f0f0 2023-12-11T20:16:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f967bcf2640 1 librados: init done 2023-12-11T20:16:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f0535724640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f0535724640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f0535724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:16:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0510005080 con 0x7f053013f0f0 2023-12-11T20:16:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f7389c56640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f7389c56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.684+0000 7f7389c56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.685+0000 7f967bcf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.685+0000 7f967bcf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.685+0000 7f967bcf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.685+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967413ffe0 con 0x7f967413f0f0 2023-12-11T20:16:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81e59c640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff82006f970 0x7ff82013e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7f738ac58640 10 monclient(hunting): get_auth_request con 0x7f738413acd0 auth_method 0 2023-12-11T20:16:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7f738ac58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7f738ac58640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7f738ac58640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2023-12-11T20:16:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 1 -- 172.21.15.86:0/1503599004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff82006f970 msgr2=0x7ff82013e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff82006f970 0x7ff82013e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 1 -- 172.21.15.86:0/1503599004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8200af1a0 msgr2=0x7ff8200aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8200af1a0 0x7ff8200aca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.687+0000 7ff81dd9b640 1 -- 172.21.15.86:0/1503599004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff818009a70 con 0x7ff820072580 2023-12-11T20:16:59.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2023-12-11T20:16:59.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 1 -- 172.21.15.86:0/2714356346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738413acd0 msgr2=0x7f73841da3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738413acd0 0x7f73841da3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 1 -- 172.21.15.86:0/2714356346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73840b0b80 msgr2=0x7f73840af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73840b0b80 0x7f73840af600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 1 monclient: found mon.noname-b 2023-12-11T20:16:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.692+0000 7f7389c56640 1 -- 172.21.15.86:0/2714356346 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c009b30 con 0x7f738413f5c0 2023-12-11T20:16:59.922 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.924+0000 7f986ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98701461c0 0x7f9870137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.922 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.924+0000 7f986ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98701461c0 0x7f9870137c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34992/0 (socket says 172.21.15.86:34992) 2023-12-11T20:16:59.922 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.924+0000 7f986ffff640 1 -- 172.21.15.86:0/3956513966 learned_addr learned my addr 172.21.15.86:0/3956513966 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:16:59.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.924+0000 7f986ffff640 10 monclient(hunting): get_auth_request con 0x7f98701461c0 auth_method 0 2023-12-11T20:16:59.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.924+0000 7f986ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:16:59.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.924+0000 7f986ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:16:59.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.924+0000 7f986ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:16:59.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:16:59.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:16:59.924 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:16:59.925 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 1 -- 172.21.15.86:0/3937837914 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0010a30 con 0x7f74ac13d3b0 2023-12-11T20:16:59.925 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.925 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:16:59.925 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: dump: 2023-12-11T20:16:59.925 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.925 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 1 -- 172.21.15.86:0/3937837914 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a0010b90 con 0x7f74ac13d3b0 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f74b1a90640 5 monclient: authenticate success, global_id 4314 2023-12-11T20:16:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f74b1a90640 10 monclient: _renew_subs 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f74b1a90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac139c50 con 0x7f74ac13d3b0 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 1 -- 172.21.15.86:0/3937837914 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0018440 con 0x7f74ac13d3b0 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f7493fff640 10 monclient: dump: 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.927 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.926+0000 7f74a9ffb640 4 set_mon_vals no callback set 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.927+0000 7f74b1a90640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.927+0000 7f74b1a90640 10 monclient: _renew_subs 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.927+0000 7f74b1a90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.927+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac139510 con 0x7f74ac13d3b0 2023-12-11T20:16:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.927+0000 7f74b1a90640 1 librados: init done 2023-12-11T20:16:59.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.928+0000 7f74b1a90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.928+0000 7f74b1a90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.928+0000 7f74b1a90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.928+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac04e840 con 0x7f74ac13d3b0 2023-12-11T20:16:59.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 1 -- 172.21.15.86:0/3956513966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98700a8a20 msgr2=0x7f98701432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98700a8a20 0x7f98701432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 1 -- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f987013f0f0 msgr2=0x7f9870143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f987013f0f0 0x7f9870143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 10 monclient: _finish_hunting 0 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 1 monclient: found mon.noname-a 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:16:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.929+0000 7f986ffff640 1 -- 172.21.15.86:0/3956513966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98640094d0 con 0x7f98701461c0 2023-12-11T20:16:59.931 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.931+0000 7f7389c56640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f738413f5c0 0x7f73840aebb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f737400c440 tx=0x7f737400c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.931 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.931+0000 7f737affd640 10 client.?.objecter ms_handle_connect 0x7f738413f5c0 2023-12-11T20:16:59.931 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.931+0000 7f737affd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.931 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.932+0000 7f966a7fc640 1 -- 172.21.15.86:0/3875813906 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9664007bb0 con 0x7f967413f0f0 2023-12-11T20:16:59.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.932+0000 7f966a7fc640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f965806c870 0x7f965806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.932+0000 7f9679a8e640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f965806c870 0x7f965806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.932+0000 7f9679a8e640 10 monclient: get_auth_request con 0x7f965806c870 auth_method 0 2023-12-11T20:16:59.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.933+0000 7f966a7fc640 10 client.4384.objecter ms_handle_connect 0x7f965806c870 2023-12-11T20:16:59.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.933+0000 7f9679a8e640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f965806c870 0x7f965806ed30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9670001560 tx=0x7f9670007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.935+0000 7f966a7fc640 1 -- 172.21.15.86:0/3875813906 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9664058e70 con 0x7f967413f0f0 2023-12-11T20:16:59.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.935+0000 7f966a7fc640 10 client.4384.objecter ms_dispatch 0x7f9674000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.935+0000 7f966a7fc640 3 client.4384.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.935+0000 7f966a7fc640 3 client.4384.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.935+0000 7f966a7fc640 20 client.4384.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.939+0000 7fa6d5ffb640 1 -- 172.21.15.86:0/1520532875 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa6d004e440 con 0x7fa6e006c3e0 2023-12-11T20:16:59.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.939+0000 7fa6d5ffb640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa6c406c870 0x7fa6c406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:16:59.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.939+0000 7fa6e4d7d640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa6c406c870 0x7fa6c406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:16:59.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.939+0000 7fa6e4d7d640 10 monclient: get_auth_request con 0x7fa6c406c870 auth_method 0 2023-12-11T20:16:59.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.940+0000 7fa6e4d7d640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa6c406c870 0x7fa6c406ed30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6e013bdf0 tx=0x7fa6dc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.938 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.940+0000 7fa6d5ffb640 10 client.4298.objecter ms_handle_connect 0x7fa6c406c870 2023-12-11T20:16:59.943 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.946+0000 7f966a7fc640 1 -- 172.21.15.86:0/3875813906 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9664054ab0 con 0x7f967413f0f0 2023-12-11T20:16:59.943 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.946+0000 7f966a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.944 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.946+0000 7f966a7fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:16:59.944 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.946+0000 7ff81dd9b640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff82014ff60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff81400c450 tx=0x7ff81400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:16:59.944 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.946+0000 7ff80effd640 10 client.?.objecter ms_handle_connect 0x7ff820072580 2023-12-11T20:16:59.944 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.946+0000 7ff80effd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 1 -- 172.21.15.86:0/2714356346 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374005480 con 0x7f738413f5c0 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: dump: 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: _finish_auth 0 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: _check_auth_tickets 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 1 -- 172.21.15.86:0/2714356346 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73740035f0 con 0x7f738413f5c0 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 5 monclient: authenticate success, global_id 4390 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 10 monclient: _renew_subs 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73840af150 con 0x7f738413f5c0 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 1 -- 172.21.15.86:0/2714356346 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73740039c0 con 0x7f738413f5c0 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f737affd640 10 monclient: dump: 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:16:59.951 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:16:59.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:16:59.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 10 monclient: _renew_subs 2023-12-11T20:16:59.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738413b180 con 0x7f738413f5c0 2023-12-11T20:16:59.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f7388c54640 4 set_mon_vals no callback set 2023-12-11T20:16:59.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.950+0000 7f738c6bb640 1 librados: init done 2023-12-11T20:16:59.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.951+0000 7f738c6bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.951+0000 7f738c6bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.951+0000 7f738c6bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:16:59.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.951+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738404e840 con 0x7f738413f5c0 2023-12-11T20:16:59.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.953+0000 7fa6d5ffb640 1 -- 172.21.15.86:0/1520532875 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa6d007af10 con 0x7fa6e006c3e0 2023-12-11T20:16:59.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.953+0000 7fa6d5ffb640 10 client.4298.objecter ms_dispatch 0x7fa6e0000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:16:59.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.953+0000 7fa6d5ffb640 3 client.4298.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:16:59.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.953+0000 7fa6d5ffb640 3 client.4298.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:16:59.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.953+0000 7fa6d5ffb640 20 client.4298.objecter dump_active .. 0 homeless 2023-12-11T20:16:59.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.953+0000 7fa6d5ffb640 1 -- 172.21.15.86:0/1520532875 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6d0076b50 con 0x7fa6e006c3e0 2023-12-11T20:16:59.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.953+0000 7fa6d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:16:59.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:16:59.953+0000 7fa6d5ffb640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:00.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.137+0000 7fa6e6580640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:00.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.137+0000 7fa6e6580640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:00.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.137+0000 7fa6e6580640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:00.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.137+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa6e0142870 con 0x7fa6e006c3e0 2023-12-11T20:17:00.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.151+0000 7f967bcf2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:00.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.151+0000 7f967bcf2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:00.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.151+0000 7f967bcf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:00.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.151+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f967404e840 con 0x7f967413f0f0 2023-12-11T20:17:00.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.178+0000 7f7493fff640 1 -- 172.21.15.86:0/3937837914 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f74a00213b0 con 0x7f74ac13d3b0 2023-12-11T20:17:00.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.179+0000 7f7493fff640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f749806c870 0x7f749806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:00.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.179+0000 7f74ab7fe640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f749806c870 0x7f749806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:00.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.179+0000 7f74ab7fe640 10 monclient: get_auth_request con 0x7f749806c870 auth_method 0 2023-12-11T20:17:00.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.179+0000 7f7493fff640 10 client.4314.objecter ms_handle_connect 0x7f749806c870 2023-12-11T20:17:00.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.179+0000 7f74ab7fe640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f749806c870 0x7f749806ed30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f749c006fd0 tx=0x7f749c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:00.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.182+0000 7f986ffff640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98701461c0 0x7f9870137c70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f986800efc0 tx=0x7f986800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:00.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.182+0000 7f9857fff640 10 client.?.objecter ms_handle_connect 0x7f98701461c0 2023-12-11T20:17:00.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.182+0000 7f9857fff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:00.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.188+0000 7f737affd640 1 -- 172.21.15.86:0/2714356346 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f737401b3b0 con 0x7f738413f5c0 2023-12-11T20:17:00.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.189+0000 7f737affd640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f736806c8a0 0x7f736806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:00.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.189+0000 7f738a457640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f736806c8a0 0x7f736806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:00.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.189+0000 7f738a457640 10 monclient: get_auth_request con 0x7f736806c8a0 auth_method 0 2023-12-11T20:17:00.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.189+0000 7f737affd640 10 client.4390.objecter ms_handle_connect 0x7f736806c8a0 2023-12-11T20:17:00.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.189+0000 7f738a457640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f736806c8a0 0x7f736806ed60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f738000a810 tx=0x7f7380008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.192+0000 7f7493fff640 1 -- 172.21.15.86:0/3937837914 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f74a005c950 con 0x7f74ac13d3b0 2023-12-11T20:17:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.192+0000 7f7493fff640 10 client.4314.objecter ms_dispatch 0x7f74ac000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.192+0000 7f7493fff640 3 client.4314.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.192+0000 7f7493fff640 3 client.4314.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.192+0000 7f7493fff640 20 client.4314.objecter dump_active .. 0 homeless 2023-12-11T20:17:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.192+0000 7f7493fff640 1 -- 172.21.15.86:0/3937837914 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f74a0058590 con 0x7f74ac13d3b0 2023-12-11T20:17:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.193+0000 7f7493fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.193+0000 7f7493fff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.194+0000 7f737affd640 1 -- 172.21.15.86:0/2714356346 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f73740582b0 con 0x7f738413f5c0 2023-12-11T20:17:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.194+0000 7f737affd640 10 client.4390.objecter ms_dispatch 0x7f7384000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.194+0000 7f737affd640 3 client.4390.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.194+0000 7f737affd640 3 client.4390.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.194+0000 7f737affd640 20 client.4390.objecter dump_active .. 0 homeless 2023-12-11T20:17:00.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.209+0000 7f737affd640 1 -- 172.21.15.86:0/2714356346 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7374057980 con 0x7f738413f5c0 2023-12-11T20:17:00.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.209+0000 7f737affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.209+0000 7f737affd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:00.209 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 1 -- 172.21.15.86:0/1503599004 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8140052d0 con 0x7ff820072580 2023-12-11T20:17:00.209 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:00.209 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: dump: 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: _finish_auth 0 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: _check_auth_tickets 2023-12-11T20:17:00.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 1 -- 172.21.15.86:0/1503599004 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff814005430 con 0x7ff820072580 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 5 monclient: authenticate success, global_id 4387 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 10 monclient: _renew_subs 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82014f630 con 0x7ff820072580 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 1 -- 172.21.15.86:0/1503599004 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff814012870 con 0x7ff820072580 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff80effd640 10 monclient: dump: 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff81cd99640 4 set_mon_vals no callback set 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 10 monclient: _renew_subs 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82014f8a0 con 0x7ff820072580 2023-12-11T20:17:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.211+0000 7ff824b73640 1 librados: init done 2023-12-11T20:17:00.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.212+0000 7ff824b73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.212+0000 7ff824b73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.212+0000 7ff824b73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:00.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.212+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff820075a40 con 0x7ff820072580 2023-12-11T20:17:00.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.213+0000 7f4e6a7fc640 1 -- 172.21.15.86:0/1284314290 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4e640129d0 con 0x7f4e7406f4e0 2023-12-11T20:17:00.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.213+0000 7f4e6a7fc640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4e5806c820 0x7f4e5806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:00.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.214+0000 7f4e7978b640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4e5806c820 0x7f4e5806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:00.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.214+0000 7f4e7978b640 10 monclient: get_auth_request con 0x7f4e5806c820 auth_method 0 2023-12-11T20:17:00.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.214+0000 7f4e6a7fc640 10 client.4308.objecter ms_handle_connect 0x7f4e5806c820 2023-12-11T20:17:00.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.214+0000 7f4e7978b640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4e5806c820 0x7f4e5806ece0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006fd0 tx=0x7f4e70008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:00.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.218+0000 7f966a7fc640 1 -- 172.21.15.86:0/3875813906 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f966401e070 con 0x7f967413f0f0 2023-12-11T20:17:00.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.218+0000 7f966a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:00.216 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.218+0000 7f966a7fc640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:00.217 INFO:teuthology.orchestra.run.smithi086.stdout:60129542147 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 10 librados: watch_flush enter 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 10 librados: watch_flush exit 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 20 client.4384.objecter shutdown clearing up homeless session... 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 10 client.4384.objecter successfully canceled tick 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f965806c870 msgr2=0x7f965806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f965806c870 0x7f965806ed30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9670001560 tx=0x7f9670007990 comp rx=0 tx=0).stop 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 10 monclient: shutdown 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f967413f0f0 msgr2=0x7f9674140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f967413f0f0 0x7f9674140300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f966400c7f0 tx=0x7f966400ccb0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 shutdown_connections 2023-12-11T20:17:00.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f965806c870 0x7f965806ed30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741461c0 0x7f9674140840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f967413f0f0 0x7f9674140300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 --2- 172.21.15.86:0/3875813906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a8a20 0x7f9674144970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 >> 172.21.15.86:0/3875813906 conn(0x7f96740b7890 msgr2=0x7f9674145ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:00.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 shutdown_connections 2023-12-11T20:17:00.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 -- 172.21.15.86:0/3875813906 wait complete. 2023-12-11T20:17:00.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.221+0000 7f967bcf2640 1 librados: shutdown 2023-12-11T20:17:00.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.232+0000 7f4e6a7fc640 1 -- 172.21.15.86:0/1284314290 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4e64059150 con 0x7f4e7406f4e0 2023-12-11T20:17:00.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.232+0000 7f4e6a7fc640 10 client.4308.objecter ms_dispatch 0x7f4e74000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:00.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.232+0000 7f4e6a7fc640 3 client.4308.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:00.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.232+0000 7f4e6a7fc640 3 client.4308.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:00.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.232+0000 7f4e6a7fc640 20 client.4308.objecter dump_active .. 0 homeless 2023-12-11T20:17:00.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.232+0000 7f4e6a7fc640 1 -- 172.21.15.86:0/1284314290 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e64054d90 con 0x7f4e7406f4e0 2023-12-11T20:17:00.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.232+0000 7f4e6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.232+0000 7f4e6a7fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 1 -- 172.21.15.86:0/3956513966 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9868014480 con 0x7f98701461c0 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: dump: 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: _finish_auth 0 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: _check_auth_tickets 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 1 -- 172.21.15.86:0/3956513966 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986800f040 con 0x7f98701461c0 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 1 -- 172.21.15.86:0/3956513966 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9868014840 con 0x7f98701461c0 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9857fff640 10 monclient: dump: 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:00.234 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f986dffb640 4 set_mon_vals no callback set 2023-12-11T20:17:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 5 monclient: authenticate success, global_id 4320 2023-12-11T20:17:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 10 monclient: _renew_subs 2023-12-11T20:17:00.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:00.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987013fdb0 con 0x7f98701461c0 2023-12-11T20:17:00.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:00.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 10 monclient: _renew_subs 2023-12-11T20:17:00.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:00.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98701381b0 con 0x7f98701461c0 2023-12-11T20:17:00.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.234+0000 7f9875d05640 1 librados: init done 2023-12-11T20:17:00.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.4 2023-12-11T20:17:00.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.236+0000 7f9875d05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.236+0000 7f9875d05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.236+0000 7f9875d05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:00.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.236+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9870137500 con 0x7f98701461c0 2023-12-11T20:17:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.236+0000 7f05177fe640 1 -- 172.21.15.86:0/3252414197 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0528040cc0 con 0x7f053013f0f0 2023-12-11T20:17:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.236+0000 7f05177fe640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f051c06c820 0x7f051c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.237+0000 7f052effd640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f051c06c820 0x7f051c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.237+0000 7f052effd640 10 monclient: get_auth_request con 0x7f051c06c820 auth_method 0 2023-12-11T20:17:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.237+0000 7f05177fe640 10 client.4301.objecter ms_handle_connect 0x7f051c06c820 2023-12-11T20:17:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.237+0000 7f052effd640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f051c06c820 0x7f051c06ece0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0520000f60 tx=0x7f0520009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.265+0000 7f05177fe640 1 -- 172.21.15.86:0/3252414197 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f052807bc50 con 0x7f053013f0f0 2023-12-11T20:17:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.265+0000 7f05177fe640 10 client.4301.objecter ms_dispatch 0x7f0530000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.265+0000 7f05177fe640 3 client.4301.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.265+0000 7f05177fe640 3 client.4301.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.265+0000 7f05177fe640 20 client.4301.objecter dump_active .. 0 homeless 2023-12-11T20:17:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.265+0000 7f05177fe640 1 -- 172.21.15.86:0/3252414197 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0528077890 con 0x7f053013f0f0 2023-12-11T20:17:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.265+0000 7f05177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.265+0000 7f05177fe640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:00.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.270+0000 7fa6d5ffb640 1 -- 172.21.15.86:0/1520532875 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6d007a170 con 0x7fa6e006c3e0 2023-12-11T20:17:00.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.270+0000 7fa6d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:00.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.270+0000 7fa6d5ffb640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:00.271 INFO:teuthology.orchestra.run.smithi086.stdout:68719476741 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 10 librados: watch_flush enter 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 10 librados: watch_flush exit 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 20 client.4298.objecter shutdown clearing up homeless session... 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 10 client.4298.objecter successfully canceled tick 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa6c406c870 msgr2=0x7fa6c406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa6c406c870 0x7fa6c406ed30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6e013bdf0 tx=0x7fa6dc009040 comp rx=0 tx=0).stop 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 10 monclient: shutdown 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e006c3e0 msgr2=0x7fa6e01441e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e006c3e0 0x7fa6e01441e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00097d0 tx=0x7fa6d00046c0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 shutdown_connections 2023-12-11T20:17:00.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fa6c406c870 0x7fa6c406ed30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e0143340 0x7fa6e01df450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e0070e70 0x7fa6e01def10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 --2- 172.21.15.86:0/1520532875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e006c3e0 0x7fa6e01441e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 >> 172.21.15.86:0/1520532875 conn(0x7fa6e00b5690 msgr2=0x7fa6e00b28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:00.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.273+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 shutdown_connections 2023-12-11T20:17:00.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.274+0000 7fa6e6580640 1 -- 172.21.15.86:0/1520532875 wait complete. 2023-12-11T20:17:00.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.274+0000 7fa6e6580640 1 librados: shutdown 2023-12-11T20:17:00.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.6 2023-12-11T20:17:00.434 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.436+0000 7f738c6bb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:00.435 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.436+0000 7f738c6bb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:00.435 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.436+0000 7f738c6bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:00.435 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.436+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f738413a980 con 0x7f738413f5c0 2023-12-11T20:17:00.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.455+0000 7f4e7c1f0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:00.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.455+0000 7f4e7c1f0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:00.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.455+0000 7f4e7c1f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:00.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.455+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4e7404e840 con 0x7f4e7406f4e0 2023-12-11T20:17:00.459 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.461+0000 7ff80effd640 1 -- 172.21.15.86:0/1503599004 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff8140129d0 con 0x7ff820072580 2023-12-11T20:17:00.459 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.461+0000 7ff80effd640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff80006c870 0x7ff80006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:00.460 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.462+0000 7ff81e59c640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff80006c870 0x7ff80006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:00.460 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.462+0000 7ff81e59c640 10 monclient: get_auth_request con 0x7ff80006c870 auth_method 0 2023-12-11T20:17:00.460 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.462+0000 7ff80effd640 10 client.4387.objecter ms_handle_connect 0x7ff80006c870 2023-12-11T20:17:00.460 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.462+0000 7ff81e59c640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff80006c870 0x7ff80006ed30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8100011e0 tx=0x7ff810009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:00.464 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.466+0000 7ff80effd640 1 -- 172.21.15.86:0/1503599004 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff814058dc0 con 0x7ff820072580 2023-12-11T20:17:00.464 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.466+0000 7ff80effd640 10 client.4387.objecter ms_dispatch 0x7ff820000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:00.464 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.466+0000 7ff80effd640 3 client.4387.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:00.464 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.466+0000 7ff80effd640 3 client.4387.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:00.464 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.466+0000 7ff80effd640 20 client.4387.objecter dump_active .. 0 homeless 2023-12-11T20:17:00.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.479+0000 7ff80effd640 1 -- 172.21.15.86:0/1503599004 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff814054a00 con 0x7ff820072580 2023-12-11T20:17:00.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.479+0000 7ff80effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.479+0000 7ff80effd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:00.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.483+0000 7f737affd640 1 -- 172.21.15.86:0/2714356346 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7374017090 con 0x7f738413f5c0 2023-12-11T20:17:00.481 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.483+0000 7f737affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:00.481 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.483+0000 7f737affd640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:00.481 INFO:teuthology.orchestra.run.smithi086.stdout:55834574851 2023-12-11T20:17:00.484 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.485+0000 7f738c6bb640 10 librados: watch_flush enter 2023-12-11T20:17:00.484 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.485+0000 7f738c6bb640 10 librados: watch_flush exit 2023-12-11T20:17:00.484 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.485+0000 7f738c6bb640 20 client.4390.objecter shutdown clearing up homeless session... 2023-12-11T20:17:00.484 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.485+0000 7f738c6bb640 10 client.4390.objecter successfully canceled tick 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f736806c8a0 msgr2=0x7f736806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f736806c8a0 0x7f736806ed60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f738000a810 tx=0x7f7380008040 comp rx=0 tx=0).stop 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 10 monclient: shutdown 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f738413f5c0 msgr2=0x7f73840aebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f738413f5c0 0x7f73840aebb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f737400c440 tx=0x7f737400c900 comp rx=0 tx=0).stop 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 shutdown_connections 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f736806c8a0 0x7f736806ed60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738413acd0 0x7f73841da3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f738413f5c0 0x7f73840aebb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 --2- 172.21.15.86:0/2714356346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73840b0b80 0x7f73840af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 >> 172.21.15.86:0/2714356346 conn(0x7f73840b3ef0 msgr2=0x7f738413c110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 shutdown_connections 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 -- 172.21.15.86:0/2714356346 wait complete. 2023-12-11T20:17:00.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.486+0000 7f738c6bb640 1 librados: shutdown 2023-12-11T20:17:00.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.496+0000 7f74b1a90640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-11T20:17:00.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.496+0000 7f74b1a90640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-11T20:17:00.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.496+0000 7f74b1a90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:00.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.496+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f74ac06b870 con 0x7f74ac13d3b0 2023-12-11T20:17:00.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.5 2023-12-11T20:17:00.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.521+0000 7f9857fff640 1 -- 172.21.15.86:0/3956513966 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f98680149a0 con 0x7f98701461c0 2023-12-11T20:17:00.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.522+0000 7f9857fff640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f985c06c870 0x7f985c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:00.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.522+0000 7f986f7fe640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f985c06c870 0x7f985c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:00.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.522+0000 7f986f7fe640 10 monclient: get_auth_request con 0x7f985c06c870 auth_method 0 2023-12-11T20:17:00.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.523+0000 7f986f7fe640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f985c06c870 0x7f985c06ed30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9860000f60 tx=0x7f98600079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:00.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.523+0000 7f9857fff640 10 client.4320.objecter ms_handle_connect 0x7f985c06c870 2023-12-11T20:17:00.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.540+0000 7f9857fff640 1 -- 172.21.15.86:0/3956513966 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9868059cb0 con 0x7f98701461c0 2023-12-11T20:17:00.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.540+0000 7f9857fff640 10 client.4320.objecter ms_dispatch 0x7f9870000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:00.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.540+0000 7f9857fff640 3 client.4320.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:00.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.540+0000 7f9857fff640 3 client.4320.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:00.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.540+0000 7f9857fff640 20 client.4320.objecter dump_active .. 0 homeless 2023-12-11T20:17:00.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.540+0000 7f9857fff640 1 -- 172.21.15.86:0/3956513966 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98680558f0 con 0x7f98701461c0 2023-12-11T20:17:00.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.540+0000 7f9857fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:00.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.540+0000 7f9857fff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:00.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.544+0000 7f7493fff640 1 -- 172.21.15.86:0/3937837914 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f74a001d020 con 0x7f74ac13d3b0 2023-12-11T20:17:00.542 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.544+0000 7f7493fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-11T20:17:00.542 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.544+0000 7f7493fff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:55834574852 2023-12-11T20:17:00.547 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 10 librados: watch_flush enter 2023-12-11T20:17:00.547 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 10 librados: watch_flush exit 2023-12-11T20:17:00.547 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 20 client.4314.objecter shutdown clearing up homeless session... 2023-12-11T20:17:00.547 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 10 client.4314.objecter successfully canceled tick 2023-12-11T20:17:00.547 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f749806c870 msgr2=0x7f749806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f749806c870 0x7f749806ed30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f749c006fd0 tx=0x7f749c008040 comp rx=0 tx=0).stop 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 10 monclient: shutdown 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ac13d3b0 msgr2=0x7f74ac1d8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ac13d3b0 0x7f74ac1d8830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f74a000b3f0 tx=0x7f74a000b8b0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 shutdown_connections 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f749806c870 0x7f749806ed30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74ac1437d0 0x7f74ac138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ac13d3b0 0x7f74ac1d8830 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 --2- 172.21.15.86:0/3937837914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74ac061190 0x7f74ac1d82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 >> 172.21.15.86:0/3937837914 conn(0x7f74ac0b3e50 msgr2=0x7f74ac0b4230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 shutdown_connections 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 -- 172.21.15.86:0/3937837914 wait complete. 2023-12-11T20:17:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.547+0000 7f74b1a90640 1 librados: shutdown 2023-12-11T20:17:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.555+0000 7f0535724640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-11T20:17:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.555+0000 7f0535724640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-11T20:17:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.555+0000 7f0535724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.555+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f0510005600 con 0x7f053013f0f0 2023-12-11T20:17:00.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.7 2023-12-11T20:17:00.559 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.561+0000 7f05177fe640 1 -- 172.21.15.86:0/3252414197 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0528039020 con 0x7f053013f0f0 2023-12-11T20:17:00.559 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.561+0000 7f05177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-11T20:17:00.559 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.561+0000 7f05177fe640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:17:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 10 librados: watch_flush enter 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 10 librados: watch_flush exit 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 20 client.4301.objecter shutdown clearing up homeless session... 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 10 client.4301.objecter successfully canceled tick 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f051c06c820 msgr2=0x7f051c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f051c06c820 0x7f051c06ece0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0520000f60 tx=0x7f0520009040 comp rx=0 tx=0).stop 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 10 monclient: shutdown 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f053013f0f0 msgr2=0x7f0530138230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f053013f0f0 0x7f0530138230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0528037040 tx=0x7f0528004a40 comp rx=0 tx=0).stop 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 shutdown_connections 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f051c06c820 0x7f051c06ece0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f053013f0f0 0x7f0530138230 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05301461c0 0x7f0530144bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 --2- 172.21.15.86:0/3252414197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05300a8a20 0x7f05301d7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 >> 172.21.15.86:0/3252414197 conn(0x7f05300b7890 msgr2=0x7f053013bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 shutdown_connections 2023-12-11T20:17:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 -- 172.21.15.86:0/3252414197 wait complete. 2023-12-11T20:17:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.563+0000 7f0535724640 1 librados: shutdown 2023-12-11T20:17:00.572 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.3 2023-12-11T20:17:00.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.594+0000 7f4e6a7fc640 1 -- 172.21.15.86:0/1284314290 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e640583b0 con 0x7f4e7406f4e0 2023-12-11T20:17:00.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.594+0000 7f4e6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:00.592 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.594+0000 7f4e6a7fc640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:68719476741 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 10 librados: watch_flush enter 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 10 librados: watch_flush exit 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 20 client.4308.objecter shutdown clearing up homeless session... 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 10 client.4308.objecter successfully canceled tick 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4e5806c820 msgr2=0x7f4e5806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4e5806c820 0x7f4e5806ece0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006fd0 tx=0x7f4e70008040 comp rx=0 tx=0).stop 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 10 monclient: shutdown 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7406f4e0 msgr2=0x7f4e740a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7406f4e0 0x7f4e740a84c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400c450 tx=0x7f4e6400c910 comp rx=0 tx=0).stop 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 shutdown_connections 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4e5806c820 0x7f4e5806ece0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e74143d00 0x7f4e741443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4e740b0b80 0x7f4e740a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 --2- 172.21.15.86:0/1284314290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7406f4e0 0x7f4e740a84c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 >> 172.21.15.86:0/1284314290 conn(0x7f4e740b3ef0 msgr2=0x7f4e741435d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:00.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 shutdown_connections 2023-12-11T20:17:00.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 -- 172.21.15.86:0/1284314290 wait complete. 2023-12-11T20:17:00.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.596+0000 7f4e7c1f0640 1 librados: shutdown 2023-12-11T20:17:00.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.2 2023-12-11T20:17:00.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.688+0000 7ff824b73640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:00.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.688+0000 7ff824b73640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:00.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.688+0000 7ff824b73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:00.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.688+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff8200ad270 con 0x7ff820072580 2023-12-11T20:17:00.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.692+0000 7ff80effd640 1 -- 172.21.15.86:0/1503599004 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff814058020 con 0x7ff820072580 2023-12-11T20:17:00.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.692+0000 7ff80effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:00.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.692+0000 7ff80effd640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 10 librados: watch_flush enter 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 10 librados: watch_flush exit 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 20 client.4387.objecter shutdown clearing up homeless session... 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 10 client.4387.objecter successfully canceled tick 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff80006c870 msgr2=0x7ff80006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff80006c870 0x7ff80006ed30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8100011e0 tx=0x7ff810009040 comp rx=0 tx=0).stop 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 10 monclient: shutdown 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 msgr2=0x7ff82014ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff82014ff60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff81400c450 tx=0x7ff81400c910 comp rx=0 tx=0).stop 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 shutdown_connections 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff80006c870 0x7ff80006ed30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8200af1a0 0x7ff8200aca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff820072580 0x7ff82014ff60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 --2- 172.21.15.86:0/1503599004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff82006f970 0x7ff82013e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 >> 172.21.15.86:0/1503599004 conn(0x7ff82006abe0 msgr2=0x7ff820137c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:00.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 shutdown_connections 2023-12-11T20:17:00.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 -- 172.21.15.86:0/1503599004 wait complete. 2023-12-11T20:17:00.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.694+0000 7ff824b73640 1 librados: shutdown 2023-12-11T20:17:00.704 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.0 2023-12-11T20:17:00.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.748+0000 7f9875d05640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-11T20:17:00.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.748+0000 7f9875d05640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-11T20:17:00.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.748+0000 7f9875d05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:00.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.748+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f987014c850 con 0x7f98701461c0 2023-12-11T20:17:00.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.751+0000 7f9857fff640 1 -- 172.21.15.86:0/3956513966 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9868018070 con 0x7f98701461c0 2023-12-11T20:17:00.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.751+0000 7f9857fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-11T20:17:00.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.751+0000 7f9857fff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:68719476741 2023-12-11T20:17:00.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 10 librados: watch_flush enter 2023-12-11T20:17:00.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 10 librados: watch_flush exit 2023-12-11T20:17:00.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 20 client.4320.objecter shutdown clearing up homeless session... 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 10 client.4320.objecter successfully canceled tick 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f985c06c870 msgr2=0x7f985c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f985c06c870 0x7f985c06ed30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9860000f60 tx=0x7f98600079b0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 10 monclient: shutdown 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98701461c0 msgr2=0x7f9870137c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98701461c0 0x7f9870137c70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f986800efc0 tx=0x7f986800c490 comp rx=0 tx=0).stop 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 shutdown_connections 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f985c06c870 0x7f985c06ed30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98701461c0 0x7f9870137c70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f987013f0f0 0x7f9870143800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 --2- 172.21.15.86:0/3956513966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98700a8a20 0x7f98701432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.753+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 >> 172.21.15.86:0/3956513966 conn(0x7f98700b7890 msgr2=0x7f98700b49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.754+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 shutdown_connections 2023-12-11T20:17:00.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.754+0000 7f9875d05640 1 -- 172.21.15.86:0/3956513966 wait complete. 2023-12-11T20:17:00.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:00.754+0000 7f9875d05640 1 librados: shutdown 2023-12-11T20:17:00.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.1 2023-12-11T20:17:01.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-11T20:17:01.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-11T20:17:01.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.343+0000 7fd6b7d1d640 10 monclient: get_monmap_and_config 2023-12-11T20:17:01.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.343+0000 7fd6b7d1d640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.343+0000 7fd6b7d1d640 10 monclient: monmap: 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:01.344185+0000 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:01.344185+0000 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:01.342 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.344+0000 7fd6b7d1d640 1 Processor -- start 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.344+0000 7fd6b7d1d640 1 -- start start 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.344+0000 7fd6b7d1d640 10 monclient: init 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fd6b00a8a20 0x7fd6b00a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b00a8a20 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b0145ab0 0x7fd6b0145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b0145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd6b013f0f0 0x7fd6b013f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6b013f0f0 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b0146430 con 0x7fd6b013f0f0 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b0145620 con 0x7fd6b00a8a20 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b00aace0 con 0x7fd6b0145ab0 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b7d1d640 10 monclient(hunting): authenticate will time out at 1507.674275s 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b62ba640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd6b013f0f0 0x7fd6b013f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.345+0000 7fd6b52b8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b0145ab0 0x7fd6b0145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b5ab9640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fd6b00a8a20 0x7fd6b00a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b62ba640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd6b013f0f0 0x7fd6b013f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34996/0 (socket says 172.21.15.86:34996) 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b62ba640 1 -- 172.21.15.86:0/3349312033 learned_addr learned my addr 172.21.15.86:0/3349312033 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b62ba640 10 monclient(hunting): get_auth_request con 0x7fd6b013f0f0 auth_method 0 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b62ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b62ba640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b62ba640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b5ab9640 10 monclient(hunting): get_auth_request con 0x7fd6b00a8a20 auth_method 0 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b5ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b5ab9640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b5ab9640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b52b8640 10 monclient(hunting): get_auth_request con 0x7fd6b0145ab0 auth_method 0 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b52b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b52b8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b52b8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b5ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b5ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b5ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b52b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b52b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.346+0000 7fd6b52b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.347+0000 7fd6b62ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.347+0000 7fd6b62ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.347+0000 7fd6b62ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 1 -- 172.21.15.86:0/3349312033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b0145ab0 msgr2=0x7fd6b0145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 1 --2- 172.21.15.86:0/3349312033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b0145ab0 0x7fd6b0145ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 1 -- 172.21.15.86:0/3349312033 >> v2:172.21.15.86:3300/0 conn(0x7fd6b013f0f0 msgr2=0x7fd6b013f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 1 --2- 172.21.15.86:0/3349312033 >> v2:172.21.15.86:3300/0 conn(0x7fd6b013f0f0 0x7fd6b013f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 1 monclient: found mon.noname-c 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.348+0000 7fd6b5ab9640 1 -- 172.21.15.86:0/3349312033 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b00aafc0 con 0x7fd6b00a8a20 2023-12-11T20:17:01.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.350+0000 7fd6b5ab9640 1 --2- 172.21.15.86:0/3349312033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 0x7fd6b00a8e00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00b8700 tx=0x7fd6ac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f096d8c4bcafeca server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 1 -- 172.21.15.86:0/3349312033 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac02e070 con 0x7fd6b00a8a20 2023-12-11T20:17:01.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:01.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: dump: 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 1 -- 172.21.15.86:0/3349312033 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac0315a0 con 0x7fd6b00a8a20 2023-12-11T20:17:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 1 -- 172.21.15.86:0/3349312033 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac031930 con 0x7fd6b00a8a20 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b7d1d640 5 monclient: authenticate success, global_id 4393 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b7d1d640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b7d1d640 4 set_mon_vals no callback set 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.354+0000 7fd6b4ab7640 10 monclient: dump: 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.354 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.355 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 10 monclient: shutdown 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/3349312033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 msgr2=0x7fd6b00a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/3349312033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 0x7fd6b00a8e00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00b8700 tx=0x7fd6ac02b840 comp rx=0 tx=0).stop 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/3349312033 shutdown_connections 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/3349312033 >> v2:172.21.15.86:3300/0 conn(0x7fd6b013f0f0 0x7fd6b013f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/3349312033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b0145ab0 0x7fd6b0145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/3349312033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 0x7fd6b00a8e00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/3349312033 >> 172.21.15.86:0/3349312033 conn(0x7fd6b00b7890 msgr2=0x7fd6b00b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/3349312033 shutdown_connections 2023-12-11T20:17:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.355+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/3349312033 wait complete. 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 10 monclient: monmap: 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 1 librados: starting msgr at 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 1 librados: starting objecter 2023-12-11T20:17:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 1 Processor -- start 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 1 -- start start 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 1 librados: setting wanted keys 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 1 librados: calling monclient init 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.356+0000 7fd6b7d1d640 10 monclient: init 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 0x7fd6b00ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b00a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b013f0f0 0x7fd6b0137dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b013f0f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b0145ab0 0x7fd6b0138310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6b0145ab0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b0142eb0 con 0x7fd6b0145ab0 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b00ad160 con 0x7fd6b00a8a20 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b01429d0 con 0x7fd6b013f0f0 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b7d1d640 10 monclient(hunting): authenticate will time out at 1507.686453s 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b62ba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b0145ab0 0x7fd6b0138310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b52b8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b013f0f0 0x7fd6b0137dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.357+0000 7fd6b5ab9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 0x7fd6b00ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b0145ab0 0x7fd6b0138310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34998/0 (socket says 172.21.15.86:34998) 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 1 -- 172.21.15.86:0/2824920514 learned_addr learned my addr 172.21.15.86:0/2824920514 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 10 monclient(hunting): get_auth_request con 0x7fd6b0145ab0 auth_method 0 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b5ab9640 10 monclient(hunting): get_auth_request con 0x7fd6b00a8a20 auth_method 0 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b5ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b5ab9640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b5ab9640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b52b8640 10 monclient(hunting): get_auth_request con 0x7fd6b013f0f0 auth_method 0 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b52b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b52b8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b52b8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b62ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b5ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b5ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.358+0000 7fd6b5ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.359+0000 7fd6b52b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.359+0000 7fd6b52b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.359+0000 7fd6b52b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 1 -- 172.21.15.86:0/2824920514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b013f0f0 msgr2=0x7fd6b0137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b013f0f0 0x7fd6b0137dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 1 -- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 msgr2=0x7fd6b00ab710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 0x7fd6b00ab710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 1 monclient: found mon.noname-a 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b62ba640 1 -- 172.21.15.86:0/2824920514 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac009400 con 0x7fd6b0145ab0 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.361+0000 7fd6b5ab9640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 0x7fd6b00ab710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.363+0000 7fd6b62ba640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b0145ab0 0x7fd6b0138310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800eed0 tx=0x7fd6a800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.363+0000 7fd6a67fc640 10 client.?.objecter ms_handle_connect 0x7fd6b0145ab0 2023-12-11T20:17:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.363+0000 7fd6a67fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:01.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 1 -- 172.21.15.86:0/2824920514 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8010040 con 0x7fd6b0145ab0 2023-12-11T20:17:01.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: dump: 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 1 -- 172.21.15.86:0/2824920514 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a800bd10 con 0x7fd6b0145ab0 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 5 monclient: authenticate success, global_id 4326 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 10 monclient: _renew_subs 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b0138850 con 0x7fd6b0145ab0 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 1 -- 172.21.15.86:0/2824920514 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8003bd0 con 0x7fd6b0145ab0 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a67fc640 10 monclient: dump: 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6a7fff640 4 set_mon_vals no callback set 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 10 monclient: _renew_subs 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b00abef0 con 0x7fd6b0145ab0 2023-12-11T20:17:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.367+0000 7fd6b7d1d640 1 librados: init done 2023-12-11T20:17:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.368+0000 7fd6b7d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.368+0000 7fd6b7d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.368+0000 7fd6b7d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.368+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b0142b90 con 0x7fd6b0145ab0 2023-12-11T20:17:01.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.396+0000 7f4f4cc4f640 10 monclient: get_monmap_and_config 2023-12-11T20:17:01.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.396+0000 7f4f4cc4f640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.396+0000 7f4f4cc4f640 10 monclient: monmap: 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:01.397165+0000 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:01.397165+0000 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:01.395 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.397+0000 7f4f4cc4f640 1 Processor -- start 2023-12-11T20:17:01.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.397+0000 7f4f4cc4f640 1 -- start start 2023-12-11T20:17:01.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.397+0000 7f4f4cc4f640 10 monclient: init 2023-12-11T20:17:01.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.397+0000 7f4f4cc4f640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.397+0000 7f4f4cc4f640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-11T20:17:01.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f480a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:01.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f481461c0 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:01.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f4f480a8e00 0x7f4f4813f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f480a8e00 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f48145620 con 0x7f4f481461c0 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f480aace0 con 0x7f4f480a8a20 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f480aafc0 con 0x7f4f480a8e00 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.398+0000 7f4f4cc4f640 10 monclient(hunting): authenticate will time out at 1507.727204s 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.399+0000 7f4f4659c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f4f480a8a20 0x7f4f48145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:59954/0 (socket says 172.21.15.86:59954) 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.399+0000 7f4f4659c640 1 -- 172.21.15.86:0/2677101624 learned_addr learned my addr 172.21.15.86:0/2677101624 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.399+0000 7f4f46d9d640 1 --2- 172.21.15.86:0/2677101624 >> v2:172.21.15.133:3300/0 conn(0x7f4f480a8e00 0x7f4f4813f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.399+0000 7f4f46d9d640 10 monclient(hunting): get_auth_request con 0x7f4f480a8e00 auth_method 0 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.399+0000 7f4f46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.399+0000 7f4f46d9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.399+0000 7f4f46d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.400+0000 7f4f46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.400+0000 7f4f46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.400+0000 7f4f46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.403+0000 7f4f46d9d640 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2023-12-11T20:17:01.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.403+0000 7f4f46d9d640 1 -- 172.21.15.86:0/2677101624 >> v1:172.21.15.133:6789/0 conn(0x7f4f480a8a20 legacy=0x7f4f48145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.403+0000 7f4f46d9d640 1 -- 172.21.15.86:0/2677101624 >> v1:172.21.15.86:6789/0 conn(0x7f4f481461c0 legacy=0x7f4f4813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.403+0000 7f4f46d9d640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.403+0000 7f4f46d9d640 1 monclient: found mon.noname-c 2023-12-11T20:17:01.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.403+0000 7f4f46d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.403+0000 7f4f46d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:17:01.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.403+0000 7f4f46d9d640 1 -- 172.21.15.86:0/2677101624 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f480b2210 con 0x7f4f480a8e00 2023-12-11T20:17:01.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.407+0000 7f4f46d9d640 1 --2- 172.21.15.86:0/2677101624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8e00 0x7f4f4813f800 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009d90 tx=0x7f4f4002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94acdb916e55509b server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 1 -- 172.21.15.86:0/2677101624 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4002cc60 con 0x7f4f480a8e00 2023-12-11T20:17:01.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:01.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 10 monclient: dump: 2023-12-11T20:17:01.409 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.409 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 1 -- 172.21.15.86:0/2677101624 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4002cdc0 con 0x7f4f480a8e00 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 1 -- 172.21.15.86:0/2677101624 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40034660 con 0x7f4f480a8e00 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.410+0000 7f4f45d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f45d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f45d9b640 10 monclient: dump: 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.411 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 5 monclient: authenticate success, global_id 4399 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 4 set_mon_vals no callback set 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 10 monclient: shutdown 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/2677101624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8e00 msgr2=0x7f4f4813f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 1 --2- 172.21.15.86:0/2677101624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8e00 0x7f4f4813f800 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009d90 tx=0x7f4f4002beb0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/2677101624 shutdown_connections 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 1 --2- 172.21.15.86:0/2677101624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8e00 0x7f4f4813f800 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/2677101624 >> 172.21.15.86:0/2677101624 conn(0x7f4f480b7890 msgr2=0x7f4f480b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/2677101624 shutdown_connections 2023-12-11T20:17:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.411+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/2677101624 wait complete. 2023-12-11T20:17:01.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 10 monclient: monmap: 2023-12-11T20:17:01.415 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.415 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:01.415 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:01.415 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:01.415 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.415 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 1 librados: starting msgr at 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 1 librados: starting objecter 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 1 Processor -- start 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 1 -- start start 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 1 librados: setting wanted keys 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 1 librados: calling monclient init 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.412+0000 7f4f4cc4f640 10 monclient: init 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8a20 0x7f4f480b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient(hunting): picked mon.noname-b con 0x7f4f480a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f480a8e00 0x7f4f48138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f480a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f481461c0 0x7f4f48140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient(hunting): picked mon.noname-a con 0x7f4f481461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f48140880 con 0x7f4f481461c0 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f48140ad0 con 0x7f4f480a8a20 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f481384a0 con 0x7f4f480a8e00 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4cc4f640 10 monclient(hunting): authenticate will time out at 1507.742377s 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f4659c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8a20 0x7f4f480b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.413+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f480a8e00 0x7f4f48138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8a20 0x7f4f480b6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52424/0 (socket says 172.21.15.86:52424) 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 1 -- 172.21.15.86:0/67367539 learned_addr learned my addr 172.21.15.86:0/67367539 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 10 monclient(hunting): get_auth_request con 0x7f4f480a8a20 auth_method 0 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f3ffff640 10 monclient(hunting): get_auth_request con 0x7f4f480a8e00 auth_method 0 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f3ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f3ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.414+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 1 -- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8a20 msgr2=0x7f4f480b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8a20 0x7f4f480b6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 1 -- 172.21.15.86:0/67367539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f481461c0 msgr2=0x7f4f48140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f481461c0 0x7f4f48140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 1 monclient: found mon.noname-c 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.417+0000 7f4f3ffff640 1 -- 172.21.15.86:0/67367539 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f40009a70 con 0x7f4f480a8e00 2023-12-11T20:17:01.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.419+0000 7f4f3ffff640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f480a8e00 0x7f4f48138cc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000b3e0 tx=0x7f4f3000b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.419+0000 7f4f3effd640 10 client.?.objecter ms_handle_connect 0x7f4f480a8e00 2023-12-11T20:17:01.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.419+0000 7f4f3effd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:01.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 1 -- 172.21.15.86:0/67367539 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30011830 con 0x7f4f480a8e00 2023-12-11T20:17:01.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: dump: 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.422 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 1 -- 172.21.15.86:0/67367539 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f30011990 con 0x7f4f480a8e00 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 5 monclient: authenticate success, global_id 4316 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 10 monclient: _renew_subs 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f48139310 con 0x7f4f480a8e00 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 1 -- 172.21.15.86:0/67367539 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30011d40 con 0x7f4f480a8e00 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f3effd640 10 monclient: dump: 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4559a640 4 set_mon_vals no callback set 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 10 monclient: _renew_subs 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f48141430 con 0x7f4f480a8e00 2023-12-11T20:17:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.423+0000 7f4f4cc4f640 1 librados: init done 2023-12-11T20:17:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.424+0000 7f4f4cc4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.424+0000 7f4f4cc4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.424+0000 7f4f4cc4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.424+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f481416b0 con 0x7f4f480a8e00 2023-12-11T20:17:01.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-11T20:17:01.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-11T20:17:01.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-11T20:17:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.605+0000 7f8aad690640 10 monclient: get_monmap_and_config 2023-12-11T20:17:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.605+0000 7f8aad690640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.606+0000 7f8aad690640 10 monclient: monmap: 2023-12-11T20:17:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:01.606506+0000 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:01.606506+0000 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.606+0000 7f8aad690640 1 Processor -- start 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.606+0000 7f8aad690640 1 -- start start 2023-12-11T20:17:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.606+0000 7f8aad690640 10 monclient: init 2023-12-11T20:17:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-12-11T20:17:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8aa80a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa81461c0 0x7f8aa80a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient(hunting): picked mon.noname-b con 0x7f8aa81461c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8aa813f0f0 0x7f8aa813f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient(hunting): picked mon.noname-a con 0x7f8aa813f0f0 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa8145620 con 0x7f8aa80a8a20 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa80aace0 con 0x7f8aa813f0f0 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa80aafc0 con 0x7f8aa81461c0 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.607+0000 7f8aad690640 10 monclient(hunting): authenticate will time out at 1507.936446s 2023-12-11T20:17:01.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-11T20:17:01.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.636+0000 7fd6a67fc640 1 -- 172.21.15.86:0/2824920514 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd6a8003d30 con 0x7fd6b0145ab0 2023-12-11T20:17:01.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.637+0000 7fd6a67fc640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd69406c870 0x7fd69406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.637+0000 7fd6b5ab9640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd69406c870 0x7fd69406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.637+0000 7fd6b5ab9640 10 monclient: get_auth_request con 0x7fd69406c870 auth_method 0 2023-12-11T20:17:01.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.638+0000 7fd6a67fc640 10 client.4326.objecter ms_handle_connect 0x7fd69406c870 2023-12-11T20:17:01.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.638+0000 7fd6b5ab9640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd69406c870 0x7fd69406ed30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac036000 tx=0x7fd6ac03d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.667+0000 7f34ce52c640 10 monclient: get_monmap_and_config 2023-12-11T20:17:01.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.667+0000 7f34ce52c640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.667+0000 7f34ce52c640 10 monclient: monmap: 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:01.668159+0000 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:01.668159+0000 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:01.666 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.668+0000 7f34ce52c640 1 Processor -- start 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.668+0000 7f34ce52c640 1 -- start start 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.668+0000 7f34ce52c640 10 monclient: init 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34c80a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34c81461c0 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34c80a8e00 0x7f34c813f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 10 monclient(hunting): picked mon.noname-b con 0x7f34c80a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c8145620 con 0x7f34c80a8a20 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c80aace0 con 0x7f34c81461c0 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c80aafc0 con 0x7f34c80a8e00 2023-12-11T20:17:01.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34ce52c640 10 monclient(hunting): authenticate will time out at 1507.998349s 2023-12-11T20:17:01.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34c77fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f34c81461c0 0x7f34c813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:59970/0 (socket says 172.21.15.86:59970) 2023-12-11T20:17:01.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.669+0000 7f34c77fe640 1 -- 172.21.15.86:0/128781666 learned_addr learned my addr 172.21.15.86:0/128781666 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:17:01.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589017469 0 0) 0x7f34c80aace0 con 0x7f34c81461c0 2023-12-11T20:17:01.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7f34c6ffd640 10 monclient(hunting): _negotiate 2023-12-11T20:17:01.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7f34c6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7f34c6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7f34c6ffd640 10 monclient(hunting): my global_id is 4408 2023-12-11T20:17:01.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b0003520 con 0x7f34c81461c0 2023-12-11T20:17:01.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7fd6a67fc640 1 -- 172.21.15.86:0/2824920514 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd6a8058850 con 0x7fd6b0145ab0 2023-12-11T20:17:01.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7fd6a67fc640 10 client.4326.objecter ms_dispatch 0x7fd6b0000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:01.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7fd6a67fc640 3 client.4326.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:01.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7fd6a67fc640 3 client.4326.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:01.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.672+0000 7fd6a67fc640 20 client.4326.objecter dump_active .. 0 homeless 2023-12-11T20:17:01.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.673+0000 7fd6a67fc640 1 -- 172.21.15.86:0/2824920514 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6a8054b90 con 0x7fd6b0145ab0 2023-12-11T20:17:01.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.673+0000 7fd6a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.672 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.673+0000 7fd6a67fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3811742347 0 0) 0x7f34b0003520 con 0x7f34c81461c0 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34c80a8e00 msgr2=0x7f34c813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34c80a8e00 0x7f34c813f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 >> v1:172.21.15.86:6789/0 conn(0x7f34c80a8a20 legacy=0x7f34c8145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 monclient: found mon.noname-c-legacy 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.133:6789/0 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c80b2210 con 0x7f34c81461c0 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1416536749 0 0) 0x7f34bc001670 con 0x7f34c81461c0 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: dump: 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.678 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] but i'm connected to v1:172.21.15.133:6789/0, reconnecting 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 >> v1:172.21.15.133:6789/0 conn(0x7f34c81461c0 legacy=0x7f34c813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34b0003970 0x7f34b0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient(hunting): picked mon.a con 0x7f34b0003970 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34c80a8e00 0x7f34b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient(hunting): picked mon.c con 0x7f34c80a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34b0004d50 0x7f34b00051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient(hunting): picked mon.b con 0x7f34b0004d50 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c80aafc0 con 0x7f34b0003970 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b000bb70 con 0x7f34b0004d50 2023-12-11T20:17:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b000bd30 con 0x7f34c80a8e00 2023-12-11T20:17:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.678+0000 7f34c6ffd640 10 monclient(hunting): _finish_auth 0 2023-12-11T20:17:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.681+0000 7f34ccac9640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34b0004d50 0x7f34b00051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.681+0000 7f34ccac9640 10 monclient(hunting): get_auth_request con 0x7f34b0004d50 auth_method 0 2023-12-11T20:17:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.681+0000 7f34ccac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.681+0000 7f34ccac9640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.681+0000 7f34ccac9640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:17:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.681+0000 7f34ccac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.681+0000 7f34ccac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.681+0000 7f34ccac9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 10 monclient(hunting): handle_auth_done global_id 4408 payload 293 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 1 -- 172.21.15.86:0/128781666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34c80a8e00 msgr2=0x7f34b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34c80a8e00 0x7f34b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 1 -- 172.21.15.86:0/128781666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34b0003970 msgr2=0x7f34b0003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34b0003970 0x7f34b0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 1 monclient: found mon.b 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:01.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 1 -- 172.21.15.86:0/128781666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c80b2210 con 0x7f34b0004d50 2023-12-11T20:17:01.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ccac9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ce52c640 5 monclient: authenticate success, global_id 4408 2023-12-11T20:17:01.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.683+0000 7f34ce52c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:01.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.684+0000 7f8aa77fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8aa813f0f0 0x7f8aa813f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.685+0000 7f34ccac9640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34b0004d50 0x7f34b00051b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009e40 tx=0x7f34b00057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3638df29c6a3bb0f server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.687+0000 7f8aa77fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8aa813f0f0 0x7f8aa813f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35012/0 (socket says 172.21.15.86:35012) 2023-12-11T20:17:01.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.687+0000 7f8aa77fe640 1 -- 172.21.15.86:0/4270408997 learned_addr learned my addr 172.21.15.86:0/4270408997 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.687+0000 7f8aa77fe640 10 monclient(hunting): get_auth_request con 0x7f8aa813f0f0 auth_method 0 2023-12-11T20:17:01.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.687+0000 7f8aa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.687+0000 7f8aa77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.687+0000 7f8aa77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.688+0000 7f8aa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.688+0000 7f8aa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.689+0000 7f8aa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c002cd70 con 0x7f34b0004d50 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 10 monclient: dump: 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.689 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c0034440 con 0x7f34b0004d50 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 1 -- 172.21.15.86:0/128781666 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0034830 con 0x7f34b0004d50 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34c6ffd640 10 monclient: dump: 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.690 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34ce52c640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.690+0000 7f34ce52c640 4 set_mon_vals no callback set 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.691+0000 7f34ce52c640 10 monclient: shutdown 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.691+0000 7f34ce52c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.691+0000 7f34ce52c640 1 -- 172.21.15.86:0/128781666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34b0004d50 msgr2=0x7f34b00051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.691+0000 7f34ce52c640 1 --2- 172.21.15.86:0/128781666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34b0004d50 0x7f34b00051b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009e40 tx=0x7f34b00057f0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.691+0000 7f34ccac9640 1 -- 172.21.15.86:0/128781666 reap_dead start 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.692+0000 7f34ce52c640 1 -- 172.21.15.86:0/128781666 shutdown_connections 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.692+0000 7f34ce52c640 1 -- 172.21.15.86:0/128781666 >> 172.21.15.86:0/128781666 conn(0x7f34c80b7890 msgr2=0x7f34c80b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.692+0000 7f34ce52c640 1 -- 172.21.15.86:0/128781666 shutdown_connections 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.692+0000 7f34ce52c640 1 -- 172.21.15.86:0/128781666 wait complete. 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.692+0000 7f34ce52c640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.692+0000 7f34ce52c640 10 monclient: monmap: 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f34ce52c640 1 librados: starting msgr at 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f34ce52c640 1 librados: starting objecter 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f34ce52c640 1 Processor -- start 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f34ce52c640 1 -- start start 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f34ce52c640 1 librados: setting wanted keys 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f34ce52c640 1 librados: calling monclient init 2023-12-11T20:17:01.694 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f34ce52c640 10 monclient: init 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 1 -- 172.21.15.86:0/4270408997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa81461c0 msgr2=0x7f8aa80a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 1 --2- 172.21.15.86:0/4270408997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa81461c0 0x7f8aa80a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 1 -- 172.21.15.86:0/4270408997 >> v1:172.21.15.86:6789/0 conn(0x7f8aa80a8a20 legacy=0x7f8aa8145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 1 monclient: found mon.noname-a 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.693+0000 7f8aa77fe640 1 -- 172.21.15.86:0/4270408997 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa813fac0 con 0x7f8aa813f0f0 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0004d50 0x7f34c8140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient(hunting): picked mon.noname-c con 0x7f34b0004d50 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34c8143330 0x7f34c8141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient(hunting): picked mon.noname-b con 0x7f34c8143330 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34c81dab00 0x7f34c81daf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient(hunting): picked mon.noname-a con 0x7f34c81dab00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c813fe20 con 0x7f34c81dab00 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.698 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81da5e0 con 0x7f34c8143330 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81418b0 con 0x7f34b0004d50 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.694+0000 7f34ce52c640 10 monclient(hunting): authenticate will time out at 1508.024060s 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.695+0000 7f34c77fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34c8143330 0x7f34c8141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34c8143330 0x7f34c8141370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52434/0 (socket says 172.21.15.86:52434) 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 1 -- 172.21.15.86:0/4220697354 learned_addr learned my addr 172.21.15.86:0/4220697354 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 10 monclient(hunting): get_auth_request con 0x7f34c8143330 auth_method 0 2023-12-11T20:17:01.699 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.701 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.696+0000 7f34c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 10 monclient(hunting): handle_auth_done global_id 4411 payload 824 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 1 -- 172.21.15.86:0/4220697354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0004d50 msgr2=0x7f34c8140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0004d50 0x7f34c8140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 1 -- 172.21.15.86:0/4220697354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34c81dab00 msgr2=0x7f34c81daf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34c81dab00 0x7f34c81daf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 1 monclient: found mon.noname-b 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.699+0000 7f34c77fe640 1 -- 172.21.15.86:0/4220697354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8141a20 con 0x7f34c8143330 2023-12-11T20:17:01.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.701+0000 7f34c77fe640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34c8143330 0x7f34c8141370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c8f0 tx=0x7f34bc00cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.702+0000 7f8aa77fe640 1 --2- 172.21.15.86:0/4270408997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa813f0f0 0x7f8aa813f580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009e50 tx=0x7f8aa002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f19e200bc2238d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.701+0000 7f34c4ff9640 10 client.?.objecter ms_handle_connect 0x7f34c8143330 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.702+0000 7f34c4ff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34c4ff9640 1 -- 172.21.15.86:0/4220697354 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc0074e0 con 0x7f34c8143330 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34c4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34c4ff9640 10 monclient: dump: 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.705 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34c4ff9640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34c4ff9640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34c4ff9640 1 -- 172.21.15.86:0/4220697354 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc007640 con 0x7f34c8143330 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34ce52c640 5 monclient: authenticate success, global_id 4411 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34ce52c640 10 monclient: _renew_subs 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34ce52c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.705+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c81d6f00 con 0x7f34c8143330 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.706+0000 7f34c4ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.706+0000 7f34c67fc640 4 set_mon_vals no callback set 2023-12-11T20:17:01.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34c4ff9640 1 -- 172.21.15.86:0/4220697354 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc004a00 con 0x7f34c8143330 2023-12-11T20:17:01.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34ce52c640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:01.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f4f3effd640 1 -- 172.21.15.86:0/67367539 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4f3001a400 con 0x7f4f480a8e00 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34c4ff9640 10 monclient: dump: 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34ce52c640 10 monclient: _renew_subs 2023-12-11T20:17:01.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34ce52c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:01.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c81d7400 con 0x7f34c8143330 2023-12-11T20:17:01.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f34ce52c640 1 librados: init done 2023-12-11T20:17:01.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.707+0000 7f4f3effd640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4f2806c870 0x7f4f2806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.708+0000 7f4f4659c640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4f2806c870 0x7f4f2806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.708+0000 7f4f4659c640 10 monclient: get_auth_request con 0x7f4f2806c870 auth_method 0 2023-12-11T20:17:01.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.708+0000 7f4f3effd640 10 client.4316.objecter ms_handle_connect 0x7f4f2806c870 2023-12-11T20:17:01.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.708+0000 7f4f4659c640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4f2806c870 0x7f4f2806ed30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005880 tx=0x7f4f38009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.710 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.708+0000 7f34ce52c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.710 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.708+0000 7f34ce52c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.710 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.708+0000 7f34ce52c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:01.710 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.708+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c81da440 con 0x7f34c8143330 2023-12-11T20:17:01.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 1 -- 172.21.15.86:0/4270408997 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa002cb00 con 0x7f8aa813f0f0 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: dump: 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.714 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 1 -- 172.21.15.86:0/4270408997 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa002cc60 con 0x7f8aa813f0f0 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aad690640 5 monclient: authenticate success, global_id 4329 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 1 -- 172.21.15.86:0/4270408997 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa00343f0 con 0x7f8aa813f0f0 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aa5ffb640 10 monclient: dump: 2023-12-11T20:17:01.715 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aad690640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.714+0000 7f8aad690640 4 set_mon_vals no callback set 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.715+0000 7f8aad690640 10 monclient: shutdown 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.715+0000 7f8aad690640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.715+0000 7f8aad690640 1 -- 172.21.15.86:0/4270408997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa813f0f0 msgr2=0x7f8aa813f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.716 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.715+0000 7f8aad690640 1 --2- 172.21.15.86:0/4270408997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa813f0f0 0x7f8aa813f580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009e50 tx=0x7f8aa002bf60 comp rx=0 tx=0).stop 2023-12-11T20:17:01.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.716+0000 7f8aad690640 1 -- 172.21.15.86:0/4270408997 shutdown_connections 2023-12-11T20:17:01.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.716+0000 7f8aad690640 1 --2- 172.21.15.86:0/4270408997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa813f0f0 0x7f8aa813f580 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.716+0000 7f8aad690640 1 --2- 172.21.15.86:0/4270408997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa81461c0 0x7f8aa80a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.716+0000 7f8aad690640 1 -- 172.21.15.86:0/4270408997 >> 172.21.15.86:0/4270408997 conn(0x7f8aa80b7890 msgr2=0x7f8aa80b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:01.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.716+0000 7f8aad690640 1 -- 172.21.15.86:0/4270408997 shutdown_connections 2023-12-11T20:17:01.717 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.716+0000 7f8aad690640 1 -- 172.21.15.86:0/4270408997 wait complete. 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.717+0000 7f8aad690640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.717+0000 7f8aad690640 10 monclient: monmap: 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.720 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.717+0000 7f8aad690640 1 librados: starting msgr at 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.718+0000 7f8aad690640 1 librados: starting objecter 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.718+0000 7f8aad690640 1 Processor -- start 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.718+0000 7f8aad690640 1 -- start start 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.718+0000 7f8aad690640 1 librados: setting wanted keys 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.718+0000 7f8aad690640 1 librados: calling monclient init 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.718+0000 7f8aad690640 10 monclient: init 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.719+0000 7f8aad690640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.719+0000 7f8aad690640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.719+0000 7f8aad690640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa80a8a20 0x7f8aa8135050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.719+0000 7f8aad690640 10 monclient(hunting): picked mon.noname-c con 0x7f8aa80a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.720+0000 7f8aad690640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8aa813f0f0 0x7f8aa8139060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.720+0000 7f8aad690640 10 monclient(hunting): picked mon.noname-b con 0x7f8aa813f0f0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.720+0000 7f8aad690640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa81461c0 0x7f8aa8135650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.720+0000 7f8aad690640 10 monclient(hunting): picked mon.noname-a con 0x7f8aa81461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aad690640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aad690640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa8138050 con 0x7f8aa81461c0 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aad690640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aad690640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa8135b90 con 0x7f8aa813f0f0 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aad690640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aad690640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa8138c10 con 0x7f8aa80a8a20 2023-12-11T20:17:01.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aad690640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aad690640 10 monclient(hunting): authenticate will time out at 1508.050301s 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aa77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa81461c0 0x7f8aa8135650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aa77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa81461c0 0x7f8aa8135650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35030/0 (socket says 172.21.15.86:35030) 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aa77fe640 1 -- 172.21.15.86:0/3346972646 learned_addr learned my addr 172.21.15.86:0/3346972646 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aa77fe640 10 monclient(hunting): get_auth_request con 0x7f8aa81461c0 auth_method 0 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aa77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.721+0000 7f8aa77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.723+0000 7f8aa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.723+0000 7f8aa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.723+0000 7f8aa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.726+0000 7f34c4ff9640 1 -- 172.21.15.86:0/4220697354 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f34bc00a5e0 con 0x7f34c8143330 2023-12-11T20:17:01.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.726+0000 7f34c4ff9640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f34b406c820 0x7f34b406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.726+0000 7f34c7fff640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f34b406c820 0x7f34b406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.727+0000 7f34c7fff640 10 monclient: get_auth_request con 0x7f34b406c820 auth_method 0 2023-12-11T20:17:01.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.727+0000 7f34c4ff9640 10 client.4411.objecter ms_handle_connect 0x7f34b406c820 2023-12-11T20:17:01.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.727+0000 7f34c7fff640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f34b406c820 0x7f34b406ece0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f34b80014e0 tx=0x7f34b8007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 1 -- 172.21.15.86:0/3346972646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa80a8a20 msgr2=0x7f8aa8135050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa80a8a20 0x7f8aa8135050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 1 -- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8aa813f0f0 msgr2=0x7f8aa8139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8aa813f0f0 0x7f8aa8139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 1 monclient: found mon.noname-a 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.728+0000 7f8aa77fe640 1 -- 172.21.15.86:0/3346972646 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa0009b30 con 0x7f8aa81461c0 2023-12-11T20:17:01.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.729+0000 7ff878d97640 10 monclient: get_monmap_and_config 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.729+0000 7ff878d97640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.729+0000 7ff878d97640 10 monclient: monmap: 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:01.730436+0000 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:01.730436+0000 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:01.728 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.730+0000 7ff878d97640 1 Processor -- start 2023-12-11T20:17:01.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.730+0000 7ff878d97640 1 -- start start 2023-12-11T20:17:01.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.730+0000 7ff878d97640 10 monclient: init 2023-12-11T20:17:01.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7f4f3effd640 1 -- 172.21.15.86:0/67367539 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4f30059100 con 0x7f4f480a8e00 2023-12-11T20:17:01.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7f4f3effd640 10 client.4316.objecter ms_dispatch 0x7f4f48000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:01.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7f4f3effd640 3 client.4316.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:01.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7f4f3effd640 3 client.4316.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:01.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7f4f3effd640 20 client.4316.objecter dump_active .. 0 homeless 2023-12-11T20:17:01.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7f4f3effd640 1 -- 172.21.15.86:0/67367539 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f30054d40 con 0x7f4f480a8e00 2023-12-11T20:17:01.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7f4f3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7f4f3effd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:01.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7ff8740a8a20 0x7ff8740a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient(hunting): picked mon.noname-c con 0x7ff8740a8a20 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff874145ab0 0x7ff874145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient(hunting): picked mon.noname-a con 0x7ff874145ab0 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff87413f0f0 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff874146430 con 0x7ff87413f0f0 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff874145620 con 0x7ff874145ab0 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8740aace0 con 0x7ff8740a8a20 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.731+0000 7ff878d97640 10 monclient(hunting): authenticate will time out at 1508.060987s 2023-12-11T20:17:01.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.732+0000 7f8aa77fe640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa81461c0 0x7f8aa8135650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009e50 tx=0x7f8aa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.732+0000 7f8a8f7fe640 10 client.?.objecter ms_handle_connect 0x7f8aa81461c0 2023-12-11T20:17:01.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.732+0000 7f8a8f7fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:01.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.733+0000 7ff871d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff874145ab0 0x7ff874145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.734+0000 7ff871d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff874145ab0 0x7ff874145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35046/0 (socket says 172.21.15.86:35046) 2023-12-11T20:17:01.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.734+0000 7ff871d9b640 1 -- 172.21.15.86:0/3944614164 learned_addr learned my addr 172.21.15.86:0/3944614164 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.734+0000 7ff871d9b640 10 monclient(hunting): get_auth_request con 0x7ff874145ab0 auth_method 0 2023-12-11T20:17:01.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.734+0000 7ff871d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.734+0000 7ff871d9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.734+0000 7ff871d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 1 -- 172.21.15.86:0/3346972646 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa003e450 con 0x7f8aa81461c0 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 10 monclient: dump: 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 1 -- 172.21.15.86:0/3346972646 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa003e5d0 con 0x7f8aa81461c0 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8aad690640 5 monclient: authenticate success, global_id 4335 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8aad690640 10 monclient: _renew_subs 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8aad690640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa8138230 con 0x7f8aa81461c0 2023-12-11T20:17:01.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8a8f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8aad690640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:01.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.740+0000 7f8aa57fa640 4 set_mon_vals no callback set 2023-12-11T20:17:01.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7f8aad690640 10 monclient: _renew_subs 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7f8aad690640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa80b5d30 con 0x7f8aa81461c0 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7f8aad690640 1 librados: init done 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7f8a8f7fe640 1 -- 172.21.15.86:0/3346972646 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa003e980 con 0x7f8aa81461c0 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7f8a8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7f8a8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7f8a8f7fe640 10 monclient: dump: 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.742 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.742 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7ff871d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7ff871d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.741+0000 7ff871d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.743+0000 7f8aad690640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.743+0000 7f8aad690640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.743+0000 7f8aad690640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.743+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa8138d80 con 0x7f8aa81461c0 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 1 -- 172.21.15.86:0/3944614164 >> v2:172.21.15.133:3300/0 conn(0x7ff8740a8a20 msgr2=0x7ff8740a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 1 --2- 172.21.15.86:0/3944614164 >> v2:172.21.15.133:3300/0 conn(0x7ff8740a8a20 0x7ff8740a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 1 -- 172.21.15.86:0/3944614164 >> v1:172.21.15.133:6789/0 conn(0x7ff87413f0f0 legacy=0x7ff87413f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 1 monclient: found mon.noname-a 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.755+0000 7ff871d9b640 1 -- 172.21.15.86:0/3944614164 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8740aafc0 con 0x7ff874145ab0 2023-12-11T20:17:01.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.767+0000 7f814bf83640 10 monclient: get_monmap_and_config 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.767+0000 7f814bf83640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.767+0000 7f814bf83640 10 monclient: monmap: 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:01.767923+0000 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:01.767923+0000 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:01.766 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:01.766 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:01.766 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:01.766 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:01.766 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 1 Processor -- start 2023-12-11T20:17:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 1 -- start start 2023-12-11T20:17:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 10 monclient: init 2023-12-11T20:17:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-12-11T20:17:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81440a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f81441461c0 0x7f81440a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 10 monclient(hunting): picked mon.noname-c con 0x7f81441461c0 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.768+0000 7f814bf83640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f814413f0f0 0x7f814413f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814bf83640 10 monclient(hunting): picked mon.noname-b con 0x7f814413f0f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814bf83640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8144145620 con 0x7f81440a8a20 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814bf83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814bf83640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81440aace0 con 0x7f81441461c0 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814bf83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814bf83640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81440aafc0 con 0x7f814413f0f0 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814bf83640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814bf83640 10 monclient(hunting): authenticate will time out at 1508.098419s 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.769+0000 7f814a520640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f814413f0f0 0x7f814413f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f814413f0f0 0x7f814413f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40206/0 (socket says 172.21.15.86:40206) 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 1 -- 172.21.15.86:0/601330304 learned_addr learned my addr 172.21.15.86:0/601330304 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 10 monclient(hunting): get_auth_request con 0x7f814413f0f0 auth_method 0 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.770+0000 7f814a520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 1 -- 172.21.15.86:0/601330304 >> v2:172.21.15.133:3300/0 conn(0x7f81441461c0 msgr2=0x7f81440a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 1 --2- 172.21.15.86:0/601330304 >> v2:172.21.15.133:3300/0 conn(0x7f81441461c0 0x7f81440a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 1 -- 172.21.15.86:0/601330304 >> v1:172.21.15.133:6789/0 conn(0x7f81440a8a20 legacy=0x7f8144145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:01.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 1 monclient: found mon.noname-b 2023-12-11T20:17:01.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.772+0000 7f814a520640 1 -- 172.21.15.86:0/601330304 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814413fa90 con 0x7f814413f0f0 2023-12-11T20:17:01.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.775+0000 7f814a520640 1 --2- 172.21.15.86:0/601330304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f814413f0f0 0x7f814413f550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f813c009e50 tx=0x7f813c02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=557c8f11f338ef7d server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.782+0000 7f8a8f7fe640 1 -- 172.21.15.86:0/3346972646 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8aa003eae0 con 0x7f8aa81461c0 2023-12-11T20:17:01.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.782+0000 7f8a8f7fe640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8a9406c870 0x7f8a9406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.783+0000 7f8aa6ffd640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8a9406c870 0x7f8a9406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.783+0000 7f8aa6ffd640 10 monclient: get_auth_request con 0x7f8a9406c870 auth_method 0 2023-12-11T20:17:01.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.783+0000 7f8aa6ffd640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8a9406c870 0x7f8a9406ed30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a98000f60 tx=0x7f8a980079d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.783+0000 7f8a8f7fe640 10 client.4335.objecter ms_handle_connect 0x7f8a9406c870 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 1 -- 172.21.15.86:0/601330304 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c02ca80 con 0x7f814413f0f0 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: dump: 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.782 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 1 -- 172.21.15.86:0/601330304 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c02cbe0 con 0x7f814413f0f0 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f814bf83640 5 monclient: authenticate success, global_id 4319 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 1 -- 172.21.15.86:0/601330304 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c0343f0 con 0x7f814413f0f0 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:17:01.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f8148d1d640 10 monclient: dump: 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f814bf83640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:01.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.784+0000 7f814bf83640 4 set_mon_vals no callback set 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.785+0000 7f814bf83640 10 monclient: shutdown 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.785+0000 7f814bf83640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.785+0000 7f814bf83640 1 -- 172.21.15.86:0/601330304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f814413f0f0 msgr2=0x7f814413f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.785+0000 7f814bf83640 1 --2- 172.21.15.86:0/601330304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f814413f0f0 0x7f814413f550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f813c009e50 tx=0x7f813c02bee0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.785+0000 7f814bf83640 1 -- 172.21.15.86:0/601330304 shutdown_connections 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.785+0000 7f814bf83640 1 --2- 172.21.15.86:0/601330304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f814413f0f0 0x7f814413f550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.785+0000 7f814bf83640 1 --2- 172.21.15.86:0/601330304 >> v2:172.21.15.133:3300/0 conn(0x7f81441461c0 0x7f81440a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.785+0000 7f814bf83640 1 -- 172.21.15.86:0/601330304 >> 172.21.15.86:0/601330304 conn(0x7f81440b7890 msgr2=0x7f81440b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.786+0000 7f814bf83640 1 -- 172.21.15.86:0/601330304 shutdown_connections 2023-12-11T20:17:01.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.788+0000 7f814bf83640 1 -- 172.21.15.86:0/601330304 wait complete. 2023-12-11T20:17:01.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.788+0000 7f814bf83640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.789+0000 7f814bf83640 10 monclient: monmap: 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:01.787 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.789+0000 7f814bf83640 1 librados: starting msgr at 2023-12-11T20:17:01.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.789+0000 7f814bf83640 1 librados: starting objecter 2023-12-11T20:17:01.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.789+0000 7f814bf83640 1 Processor -- start 2023-12-11T20:17:01.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.789+0000 7f814bf83640 1 -- start start 2023-12-11T20:17:01.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.790+0000 7f814bf83640 1 librados: setting wanted keys 2023-12-11T20:17:01.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.790+0000 7f814bf83640 1 librados: calling monclient init 2023-12-11T20:17:01.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.790+0000 7f814bf83640 10 monclient: init 2023-12-11T20:17:01.789 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.791+0000 7f814bf83640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.789 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.791+0000 7f814bf83640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:17:01.789 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.791+0000 7f814bf83640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81440a8a20 0x7f8144136ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 10 monclient(hunting): picked mon.noname-c con 0x7f81440a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f814413f0f0 0x7f81441d7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 10 monclient(hunting): picked mon.noname-b con 0x7f814413f0f0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81441461c0 0x7f8144140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 10 monclient(hunting): picked mon.noname-a con 0x7f81441461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81441d7020 con 0x7f81441461c0 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81441d7300 con 0x7f814413f0f0 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8144139310 con 0x7f81440a8a20 2023-12-11T20:17:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.792+0000 7f814bf83640 10 monclient(hunting): authenticate will time out at 1508.121194s 2023-12-11T20:17:01.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.793+0000 7f8149d1f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81440a8a20 0x7f8144136ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.793+0000 7f8149d1f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81440a8a20 0x7f8144136ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40218/0 (socket says 172.21.15.86:40218) 2023-12-11T20:17:01.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.793+0000 7f8149d1f640 1 -- 172.21.15.86:0/690481854 learned_addr learned my addr 172.21.15.86:0/690481854 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:01.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.793+0000 7f8149d1f640 10 monclient(hunting): get_auth_request con 0x7f81440a8a20 auth_method 0 2023-12-11T20:17:01.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.793+0000 7f8149d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:01.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.793+0000 7f8149d1f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:01.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.793+0000 7f8149d1f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:01.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.795+0000 7f8149d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:01.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.795+0000 7f8149d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:01.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.796+0000 7f8149d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:01.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-12-11T20:17:01.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 1 -- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f814413f0f0 msgr2=0x7f81441d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f814413f0f0 0x7f81441d7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 1 -- 172.21.15.86:0/690481854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81441461c0 msgr2=0x7f8144140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:01.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81441461c0 0x7f8144140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:01.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 10 monclient: _finish_hunting 0 2023-12-11T20:17:01.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 1 monclient: found mon.noname-c 2023-12-11T20:17:01.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:01.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.803+0000 7f8149d1f640 1 -- 172.21.15.86:0/690481854 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c009b30 con 0x7f81440a8a20 2023-12-11T20:17:01.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.806+0000 7f8149d1f640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81440a8a20 0x7f8144136ec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f814000c410 tx=0x7f814000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.806+0000 7f813a7fc640 10 client.?.objecter ms_handle_connect 0x7f81440a8a20 2023-12-11T20:17:01.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.806+0000 7f813a7fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 1 -- 172.21.15.86:0/690481854 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81400035f0 con 0x7f81440a8a20 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 10 monclient: dump: 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 10 monclient: _finish_auth 0 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 1 -- 172.21.15.86:0/690481854 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8140003750 con 0x7f81440a8a20 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f814bf83640 5 monclient: authenticate success, global_id 4322 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f814bf83640 10 monclient: _renew_subs 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f814bf83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81441d7d60 con 0x7f81440a8a20 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f814bf83640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.812+0000 7f813bfff640 4 set_mon_vals no callback set 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.813+0000 7f814bf83640 10 monclient: _renew_subs 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.813+0000 7f813a7fc640 1 -- 172.21.15.86:0/690481854 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140003b40 con 0x7f81440a8a20 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.813+0000 7f814bf83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.813+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8144140740 con 0x7f81440a8a20 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.813+0000 7f814bf83640 1 librados: init done 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.813+0000 7f813a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.813+0000 7f813a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.813+0000 7f813a7fc640 10 monclient: dump: 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.814+0000 7f814bf83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.814+0000 7f814bf83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.815+0000 7f814bf83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.815+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81441d7160 con 0x7f81440a8a20 2023-12-11T20:17:01.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.841+0000 7f8a8f7fe640 1 -- 172.21.15.86:0/3346972646 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8aa007bed0 con 0x7f8aa81461c0 2023-12-11T20:17:01.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.841+0000 7f8a8f7fe640 10 client.4335.objecter ms_dispatch 0x7f8aa8000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:01.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.841+0000 7f8a8f7fe640 3 client.4335.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:01.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.841+0000 7f8a8f7fe640 3 client.4335.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:01.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.841+0000 7f8a8f7fe640 20 client.4335.objecter dump_active .. 0 homeless 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.854+0000 7faf341b0640 10 monclient: get_monmap_and_config 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.854+0000 7faf341b0640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.854+0000 7faf341b0640 10 monclient: monmap: 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:01.855449+0000 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:01.855449+0000 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:01.854 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.855+0000 7faf341b0640 1 Processor -- start 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.856+0000 7faf341b0640 1 -- start start 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.856+0000 7faf341b0640 10 monclient: init 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faf2c0a8a20 0x7faf2c0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient(hunting): picked mon.noname-a con 0x7faf2c0a8a20 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7faf2c145ab0 0x7faf2c145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient(hunting): picked mon.noname-c con 0x7faf2c145ab0 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf2c13f0f0 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2c146430 con 0x7faf2c13f0f0 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2c145620 con 0x7faf2c0a8a20 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2c0aace0 con 0x7faf2c145ab0 2023-12-11T20:17:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.857 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.857+0000 7faf341b0640 10 monclient(hunting): authenticate will time out at 1508.186243s 2023-12-11T20:17:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.866+0000 7faf31f4c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faf2c0a8a20 0x7faf2c0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.880 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.881+0000 7f54bb3f0640 10 monclient: get_monmap_and_config 2023-12-11T20:17:01.880 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.881+0000 7f54bb3f0640 10 monclient: build_initial_monmap 2023-12-11T20:17:01.880 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.881+0000 7f54bb3f0640 10 monclient: monmap: 2023-12-11T20:17:01.880 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:01.880 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:01.880 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:01.881689+0000 2023-12-11T20:17:01.880 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:01.881689+0000 2023-12-11T20:17:01.880 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.881+0000 7f54bb3f0640 1 Processor -- start 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 1 -- start start 2023-12-11T20:17:01.881 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient: init 2023-12-11T20:17:01.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:01.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-12-11T20:17:01.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54b40a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f54b41461c0 0x7f54b40a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient(hunting): picked mon.noname-a con 0x7f54b41461c0 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f54b413f0f0 0x7f54b413f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient(hunting): picked mon.noname-c con 0x7f54b413f0f0 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b4145620 con 0x7f54b40a8a20 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b40aace0 con 0x7f54b41461c0 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b40aafc0 con 0x7f54b413f0f0 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:01.883 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.882+0000 7f54bb3f0640 10 monclient(hunting): authenticate will time out at 1508.212040s 2023-12-11T20:17:01.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.888+0000 7f813a7fc640 1 -- 172.21.15.86:0/690481854 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8140003ca0 con 0x7f81440a8a20 2023-12-11T20:17:01.886 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.888+0000 7f813a7fc640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f812806c820 0x7f812806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:01.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.889+0000 7f814951e640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f812806c820 0x7f812806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:01.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.889+0000 7f814951e640 10 monclient: get_auth_request con 0x7f812806c820 auth_method 0 2023-12-11T20:17:01.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.889+0000 7f814951e640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f812806c820 0x7f812806ece0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f81440b7410 tx=0x7f8134006bf0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:01.887 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.889+0000 7f813a7fc640 10 client.4322.objecter ms_handle_connect 0x7f812806c820 2023-12-11T20:17:01.907 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.908+0000 7fd6b7d1d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:01.907 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.908+0000 7fd6b7d1d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:01.907 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.908+0000 7fd6b7d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:01.907 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.908+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd6b004e840 con 0x7fd6b0145ab0 2023-12-11T20:17:01.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.997+0000 7f4f4cc4f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:01.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.997+0000 7f4f4cc4f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:01.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.997+0000 7f4f4cc4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:01.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:01.997+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4f48138630 con 0x7f4f480a8e00 2023-12-11T20:17:01.999 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.001+0000 7faf3174b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7faf2c145ab0 0x7faf2c145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:02.000 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.002+0000 7faf3174b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7faf2c145ab0 0x7faf2c145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52482/0 (socket says 172.21.15.86:52482) 2023-12-11T20:17:02.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.002+0000 7faf3174b640 1 -- 172.21.15.86:0/3153985998 learned_addr learned my addr 172.21.15.86:0/3153985998 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:02.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.002+0000 7faf3174b640 10 monclient(hunting): get_auth_request con 0x7faf2c145ab0 auth_method 0 2023-12-11T20:17:02.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.002+0000 7faf3174b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:02.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.002+0000 7faf3174b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:02.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.002+0000 7faf3174b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:02.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7f54b998d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f54b413f0f0 0x7f54b413f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:02.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7faf3174b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:02.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7faf3174b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:02.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7faf3174b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:02.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7f54b998d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f54b413f0f0 0x7f54b413f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52490/0 (socket says 172.21.15.86:52490) 2023-12-11T20:17:02.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7f54b998d640 1 -- 172.21.15.86:0/617514223 learned_addr learned my addr 172.21.15.86:0/617514223 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:02.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7f54b998d640 10 monclient(hunting): get_auth_request con 0x7f54b413f0f0 auth_method 0 2023-12-11T20:17:02.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7f54b998d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:02.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7f54b998d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:02.003 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.003+0000 7f54b998d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:02.003 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.005+0000 7f34c4ff9640 1 -- 172.21.15.86:0/4220697354 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f34bc001d20 con 0x7f34c8143330 2023-12-11T20:17:02.003 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.005+0000 7f34c4ff9640 10 client.4411.objecter ms_dispatch 0x7f34c8000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:02.003 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.005+0000 7f34c4ff9640 3 client.4411.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:02.003 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.005+0000 7f34c4ff9640 3 client.4411.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:02.004 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.005+0000 7f54b998d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:02.004 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.005+0000 7f54b998d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:02.004 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.005+0000 7f54b998d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:02.004 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.005+0000 7f34c4ff9640 20 client.4411.objecter dump_active .. 0 homeless 2023-12-11T20:17:02.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 10 monclient(hunting): handle_auth_done global_id 4414 payload 274 2023-12-11T20:17:02.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 1 -- 172.21.15.86:0/3153985998 >> v2:172.21.15.86:3300/0 conn(0x7faf2c0a8a20 msgr2=0x7faf2c0a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:02.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 1 --2- 172.21.15.86:0/3153985998 >> v2:172.21.15.86:3300/0 conn(0x7faf2c0a8a20 0x7faf2c0a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 1 -- 172.21.15.86:0/3153985998 >> v1:172.21.15.133:6789/0 conn(0x7faf2c13f0f0 legacy=0x7faf2c13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:02.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 10 monclient: _finish_hunting 0 2023-12-11T20:17:02.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 1 monclient: found mon.noname-c 2023-12-11T20:17:02.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:02.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.016+0000 7faf3174b640 1 -- 172.21.15.86:0/3153985998 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c0aafc0 con 0x7faf2c145ab0 2023-12-11T20:17:02.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.017+0000 7f34c4ff9640 1 -- 172.21.15.86:0/4220697354 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34bc054080 con 0x7f34c8143330 2023-12-11T20:17:02.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.017+0000 7f34c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.017+0000 7f34c4ff9640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 10 monclient(hunting): handle_auth_done global_id 4417 payload 274 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 1 -- 172.21.15.86:0/617514223 >> v2:172.21.15.86:3300/0 conn(0x7f54b41461c0 msgr2=0x7f54b40a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 1 --2- 172.21.15.86:0/617514223 >> v2:172.21.15.86:3300/0 conn(0x7f54b41461c0 0x7f54b40a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 1 -- 172.21.15.86:0/617514223 >> v1:172.21.15.86:6789/0 conn(0x7f54b40a8a20 legacy=0x7f54b4145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 10 monclient: _finish_hunting 0 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 1 monclient: found mon.noname-c 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.020+0000 7f54b998d640 1 -- 172.21.15.86:0/617514223 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b413fa90 con 0x7f54b413f0f0 2023-12-11T20:17:02.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.026+0000 7faf3174b640 1 --2- 172.21.15.86:0/3153985998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c145ab0 0x7faf2c145ef0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf2c0a9a50 tx=0x7faf1c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a32bfe98ae25fb5 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.027+0000 7f54b998d640 1 --2- 172.21.15.86:0/617514223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b413f550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54b40b3990 tx=0x7f54ac02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeded4a7933def7b server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf30f4a640 1 -- 172.21.15.86:0/3153985998 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c031870 con 0x7faf2c145ab0 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf30f4a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf30f4a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf30f4a640 10 monclient: dump: 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf30f4a640 10 monclient: _finish_auth 0 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf30f4a640 10 monclient: _check_auth_tickets 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf30f4a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf341b0640 5 monclient: authenticate success, global_id 4414 2023-12-11T20:17:02.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.030+0000 7faf341b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:02.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf30f4a640 1 -- 172.21.15.86:0/3153985998 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1c0319d0 con 0x7faf2c145ab0 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf30f4a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf341b0640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf30f4a640 1 -- 172.21.15.86:0/3153985998 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c031d60 con 0x7faf2c145ab0 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf30f4a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf341b0640 4 set_mon_vals no callback set 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf30f4a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf30f4a640 10 monclient: dump: 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.034 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.034 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.034 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 1 -- 172.21.15.86:0/617514223 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac02ca80 con 0x7f54b413f0f0 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: dump: 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: _finish_auth 0 2023-12-11T20:17:02.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: _check_auth_tickets 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 1 -- 172.21.15.86:0/617514223 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac02cbe0 con 0x7f54b413f0f0 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54bb3f0640 5 monclient: authenticate success, global_id 4417 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54bb3f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54bb3f0640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54bb3f0640 4 set_mon_vals no callback set 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 1 -- 172.21.15.86:0/617514223 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0343f0 con 0x7f54b413f0f0 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7f54abfff640 10 monclient: dump: 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.036 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.037 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.037 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.037 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.037 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.037 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.037 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf341b0640 10 monclient: shutdown 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf341b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf341b0640 1 -- 172.21.15.86:0/3153985998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c145ab0 msgr2=0x7faf2c145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.035+0000 7faf341b0640 1 --2- 172.21.15.86:0/3153985998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c145ab0 0x7faf2c145ef0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf2c0a9a50 tx=0x7faf1c02b910 comp rx=0 tx=0).stop 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7faf341b0640 1 -- 172.21.15.86:0/3153985998 shutdown_connections 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7faf341b0640 1 --2- 172.21.15.86:0/3153985998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c145ab0 0x7faf2c145ef0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7faf341b0640 1 --2- 172.21.15.86:0/3153985998 >> v2:172.21.15.86:3300/0 conn(0x7faf2c0a8a20 0x7faf2c0a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7faf341b0640 1 -- 172.21.15.86:0/3153985998 >> 172.21.15.86:0/3153985998 conn(0x7faf2c0b7890 msgr2=0x7faf2c0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7faf341b0640 1 -- 172.21.15.86:0/3153985998 shutdown_connections 2023-12-11T20:17:02.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7faf341b0640 1 -- 172.21.15.86:0/3153985998 wait complete. 2023-12-11T20:17:02.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 10 monclient: shutdown 2023-12-11T20:17:02.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 1 -- 172.21.15.86:0/617514223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 msgr2=0x7f54b413f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/617514223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b413f550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54b40b3990 tx=0x7f54ac02bee0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 1 -- 172.21.15.86:0/617514223 shutdown_connections 2023-12-11T20:17:02.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/617514223 >> v2:172.21.15.86:3300/0 conn(0x7f54b41461c0 0x7f54b40a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/617514223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b413f550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 1 -- 172.21.15.86:0/617514223 >> 172.21.15.86:0/617514223 conn(0x7f54b40b7890 msgr2=0x7f54b40b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 1 -- 172.21.15.86:0/617514223 shutdown_connections 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.036+0000 7f54bb3f0640 1 -- 172.21.15.86:0/617514223 wait complete. 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 10 monclient: build_initial_monmap 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 10 monclient: monmap: 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:02.040 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 1 librados: starting msgr at 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 1 librados: starting objecter 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7f54bb3f0640 10 monclient: build_initial_monmap 2023-12-11T20:17:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7f54bb3f0640 10 monclient: monmap: 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:02.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 1 Processor -- start 2023-12-11T20:17:02.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 1 -- start start 2023-12-11T20:17:02.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 1 librados: setting wanted keys 2023-12-11T20:17:02.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 1 librados: calling monclient init 2023-12-11T20:17:02.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.037+0000 7faf341b0640 10 monclient: init 2023-12-11T20:17:02.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7f54bb3f0640 1 librados: starting msgr at 2023-12-11T20:17:02.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7f54bb3f0640 1 librados: starting objecter 2023-12-11T20:17:02.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7f54bb3f0640 1 Processor -- start 2023-12-11T20:17:02.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7f54bb3f0640 1 -- start start 2023-12-11T20:17:02.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7f54bb3f0640 1 librados: setting wanted keys 2023-12-11T20:17:02.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7f54bb3f0640 1 librados: calling monclient init 2023-12-11T20:17:02.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7f54bb3f0640 10 monclient: init 2023-12-11T20:17:02.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:02.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:17:02.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c0a8a20 0x7faf2c138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient(hunting): picked mon.noname-b con 0x7faf2c0a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf2c13f0f0 0x7faf2c144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient(hunting): picked mon.noname-a con 0x7faf2c13f0f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf2c145ab0 0x7faf2c1389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient(hunting): picked mon.noname-c con 0x7faf2c145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c137e50 con 0x7faf2c13f0f0 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:17:02.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54b40a8a20 0x7f54b40b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient(hunting): picked mon.noname-c con 0x7f54b40a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:02.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1443c0 con 0x7faf2c0a8a20 2023-12-11T20:17:02.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b41da820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient(hunting): picked mon.noname-b con 0x7f54b413f0f0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b41461c0 0x7f54b4132b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient(hunting): picked mon.noname-a con 0x7f54b41461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41db140 con 0x7f54b41461c0 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41dad60 con 0x7f54b413f0f0 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b4145150 con 0x7f54b40a8a20 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7f54bb3f0640 10 monclient(hunting): authenticate will time out at 1508.368294s 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1445d0 con 0x7faf2c145ab0 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.038+0000 7faf341b0640 10 monclient(hunting): authenticate will time out at 1508.367824s 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7faf31f4c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c0a8a20 0x7faf2c138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7faf31f4c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c0a8a20 0x7faf2c138190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52504/0 (socket says 172.21.15.86:52504) 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7faf31f4c640 1 -- 172.21.15.86:0/1993243145 learned_addr learned my addr 172.21.15.86:0/1993243145 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7faf31f4c640 10 monclient(hunting): get_auth_request con 0x7faf2c0a8a20 auth_method 0 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7faf31f4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:02.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7faf31f4c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:02.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.039+0000 7faf31f4c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:02.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.040+0000 7faf31f4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:02.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.040+0000 7faf31f4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:02.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.040+0000 7faf31f4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:02.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.040+0000 7f54b898b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b41da820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:02.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 10 monclient(hunting): handle_auth_done global_id 4420 payload 824 2023-12-11T20:17:02.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 1 -- 172.21.15.86:0/1993243145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf2c145ab0 msgr2=0x7faf2c1389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf2c145ab0 0x7faf2c1389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 1 -- 172.21.15.86:0/1993243145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf2c13f0f0 msgr2=0x7faf2c144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf2c13f0f0 0x7faf2c144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 10 monclient: _finish_hunting 0 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 1 monclient: found mon.noname-b 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.043+0000 7faf31f4c640 1 -- 172.21.15.86:0/1993243145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c0094d0 con 0x7faf2c0a8a20 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7f54b898b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b41da820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52510/0 (socket says 172.21.15.86:52510) 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7f54b898b640 1 -- 172.21.15.86:0/4120419735 learned_addr learned my addr 172.21.15.86:0/4120419735 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7f54b898b640 10 monclient(hunting): get_auth_request con 0x7f54b413f0f0 auth_method 0 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7f54b898b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7f54b898b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:02.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7f54b898b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:02.054 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7faf31f4c640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c0a8a20 0x7faf2c138190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faf2800c7f0 tx=0x7faf2800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.054 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7faf227fc640 10 client.?.objecter ms_handle_connect 0x7faf2c0a8a20 2023-12-11T20:17:02.054 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.045+0000 7faf227fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:02.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.046+0000 7f54b898b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:02.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.046+0000 7f54b898b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:02.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.046+0000 7f54b898b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:02.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2023-12-11T20:17:02.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 1 -- 172.21.15.86:0/4120419735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54b40a8a20 msgr2=0x7f54b40b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54b40a8a20 0x7f54b40b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 1 -- 172.21.15.86:0/4120419735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b41461c0 msgr2=0x7f54b4132b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b41461c0 0x7f54b4132b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 10 monclient: _finish_hunting 0 2023-12-11T20:17:02.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 1 monclient: found mon.noname-b 2023-12-11T20:17:02.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:02.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.049+0000 7f54b898b640 1 -- 172.21.15.86:0/4120419735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac009b30 con 0x7f54b413f0f0 2023-12-11T20:17:02.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.053+0000 7f54b898b640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b41da820 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b410 tx=0x7f54a400b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.053+0000 7f54a9ffb640 10 client.?.objecter ms_handle_connect 0x7f54b413f0f0 2023-12-11T20:17:02.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.053+0000 7f54a9ffb640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:02.058 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 1 -- 172.21.15.86:0/1993243145 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28004220 con 0x7faf2c0a8a20 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: dump: 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.059 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: _finish_auth 0 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: _check_auth_tickets 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 1 -- 172.21.15.86:0/1993243145 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf28004380 con 0x7faf2c0a8a20 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 1 -- 172.21.15.86:0/1993243145 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28007840 con 0x7faf2c0a8a20 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf227fc640 10 monclient: dump: 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf23fff640 4 set_mon_vals no callback set 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 5 monclient: authenticate success, global_id 4420 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 10 monclient: _renew_subs 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2c145030 con 0x7faf2c0a8a20 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 10 monclient: _renew_subs 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2c1390c0 con 0x7faf2c0a8a20 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.054+0000 7faf341b0640 1 librados: init done 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.055+0000 7faf341b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.055+0000 7faf341b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.055+0000 7faf341b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.055+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2c1447f0 con 0x7faf2c0a8a20 2023-12-11T20:17:02.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.115+0000 7ff871d9b640 1 --2- 172.21.15.86:0/3944614164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff874145ab0 0x7ff874145ef0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff8740b8700 tx=0x7ff86802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8929b342fccb729b server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.123+0000 7fd6a67fc640 1 -- 172.21.15.86:0/2824920514 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd6a801a090 con 0x7fd6b0145ab0 2023-12-11T20:17:02.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.123+0000 7fd6a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:02.121 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.123+0000 7fd6a67fc640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:60129542147 2023-12-11T20:17:02.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.125+0000 7fd6b7d1d640 10 librados: watch_flush enter 2023-12-11T20:17:02.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.125+0000 7f813a7fc640 1 -- 172.21.15.86:0/690481854 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f81400583d0 con 0x7f81440a8a20 2023-12-11T20:17:02.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.125+0000 7f813a7fc640 10 client.4322.objecter ms_dispatch 0x7f8144000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:02.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.125+0000 7f813a7fc640 3 client.4322.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:02.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.125+0000 7f813a7fc640 3 client.4322.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:02.124 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 10 librados: watch_flush exit 2023-12-11T20:17:02.125 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 20 client.4326.objecter shutdown clearing up homeless session... 2023-12-11T20:17:02.125 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 10 client.4326.objecter successfully canceled tick 2023-12-11T20:17:02.125 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7f813a7fc640 20 client.4322.objecter dump_active .. 0 homeless 2023-12-11T20:17:02.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd69406c870 msgr2=0x7fd69406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd69406c870 0x7fd69406ed30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac036000 tx=0x7fd6ac03d040 comp rx=0 tx=0).stop 2023-12-11T20:17:02.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 10 monclient: shutdown 2023-12-11T20:17:02.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b0145ab0 msgr2=0x7fd6b0138310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b0145ab0 0x7fd6b0138310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800eed0 tx=0x7fd6a800b480 comp rx=0 tx=0).stop 2023-12-11T20:17:02.126 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 shutdown_connections 2023-12-11T20:17:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd69406c870 0x7fd69406ed30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b013f0f0 0x7fd6b0137dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b0145ab0 0x7fd6b0138310 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 --2- 172.21.15.86:0/2824920514 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6b00a8a20 0x7fd6b00ab710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 >> 172.21.15.86:0/2824920514 conn(0x7fd6b00b7890 msgr2=0x7fd6b00b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 shutdown_connections 2023-12-11T20:17:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.126+0000 7fd6b7d1d640 1 -- 172.21.15.86:0/2824920514 wait complete. 2023-12-11T20:17:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.127+0000 7fd6b7d1d640 1 librados: shutdown 2023-12-11T20:17:02.139 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.4 2023-12-11T20:17:02.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.145+0000 7f813a7fc640 1 -- 172.21.15.86:0/690481854 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8140054010 con 0x7f81440a8a20 2023-12-11T20:17:02.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.145+0000 7f813a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.145+0000 7f813a7fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:02.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.149+0000 7f8a8f7fe640 1 -- 172.21.15.86:0/3346972646 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8aa0077b10 con 0x7f8aa81461c0 2023-12-11T20:17:02.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.149+0000 7f8a8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.149+0000 7f8a8f7fe640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:02.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 1 -- 172.21.15.86:0/3944614164 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868031870 con 0x7ff874145ab0 2023-12-11T20:17:02.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:02.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: dump: 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: _finish_auth 0 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: _check_auth_tickets 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:02.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 1 -- 172.21.15.86:0/3944614164 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8680319d0 con 0x7ff874145ab0 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 1 -- 172.21.15.86:0/3944614164 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868031d60 con 0x7ff874145ab0 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff87159a640 10 monclient: dump: 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.155 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff878d97640 5 monclient: authenticate success, global_id 4338 2023-12-11T20:17:02.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff878d97640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:02.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.155+0000 7ff878d97640 4 set_mon_vals no callback set 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 10 monclient: shutdown 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 1 -- 172.21.15.86:0/3944614164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff874145ab0 msgr2=0x7ff874145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 1 --2- 172.21.15.86:0/3944614164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff874145ab0 0x7ff874145ef0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff8740b8700 tx=0x7ff86802b910 comp rx=0 tx=0).stop 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 1 -- 172.21.15.86:0/3944614164 shutdown_connections 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 1 --2- 172.21.15.86:0/3944614164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff874145ab0 0x7ff874145ef0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 1 --2- 172.21.15.86:0/3944614164 >> v2:172.21.15.133:3300/0 conn(0x7ff8740a8a20 0x7ff8740a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 1 -- 172.21.15.86:0/3944614164 >> 172.21.15.86:0/3944614164 conn(0x7ff8740b7890 msgr2=0x7ff8740b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 1 -- 172.21.15.86:0/3944614164 shutdown_connections 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.156+0000 7ff878d97640 1 -- 172.21.15.86:0/3944614164 wait complete. 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.157+0000 7ff878d97640 10 monclient: build_initial_monmap 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.157+0000 7ff878d97640 10 monclient: monmap: 2023-12-11T20:17:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.158+0000 7ff878d97640 1 librados: starting msgr at 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.158+0000 7ff878d97640 1 librados: starting objecter 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.158+0000 7ff878d97640 1 Processor -- start 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.158+0000 7ff878d97640 1 -- start start 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.158+0000 7ff878d97640 1 librados: setting wanted keys 2023-12-11T20:17:02.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.158+0000 7ff878d97640 1 librados: calling monclient init 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.158+0000 7ff878d97640 10 monclient: init 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8740a8a20 0x7ff8740b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient(hunting): picked mon.noname-b con 0x7ff8740a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff87413f0f0 0x7ff874136ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient(hunting): picked mon.noname-a con 0x7ff87413f0f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff874145ab0 0x7ff874137bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient(hunting): picked mon.noname-c con 0x7ff874145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff874137590 con 0x7ff87413f0f0 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff874132950 con 0x7ff8740a8a20 2023-12-11T20:17:02.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:02.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff874132b10 con 0x7ff874145ab0 2023-12-11T20:17:02.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:02.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.159+0000 7ff878d97640 10 monclient(hunting): authenticate will time out at 1508.488622s 2023-12-11T20:17:02.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.161+0000 7ff871d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff87413f0f0 0x7ff874136ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff87413f0f0 0x7ff874136ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35106/0 (socket says 172.21.15.86:35106) 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 1 -- 172.21.15.86:0/648304599 learned_addr learned my addr 172.21.15.86:0/648304599 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 10 monclient(hunting): get_auth_request con 0x7ff87413f0f0 auth_method 0 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:02.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.162+0000 7ff871d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:02.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2023-12-11T20:17:02.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 1 -- 172.21.15.86:0/648304599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff874145ab0 msgr2=0x7ff874137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff874145ab0 0x7ff874137bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 1 -- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8740a8a20 msgr2=0x7ff8740b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8740a8a20 0x7ff8740b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 10 monclient: _finish_hunting 0 2023-12-11T20:17:02.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 1 monclient: found mon.noname-a 2023-12-11T20:17:02.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:02.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:02.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.165+0000 7ff871d9b640 1 -- 172.21.15.86:0/648304599 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8680094d0 con 0x7ff87413f0f0 2023-12-11T20:17:02.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.168+0000 7ff871d9b640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff87413f0f0 0x7ff874136ec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff8680090b0 tx=0x7ff8680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.168+0000 7ff862ffd640 10 client.?.objecter ms_handle_connect 0x7ff87413f0f0 2023-12-11T20:17:02.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.168+0000 7ff862ffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:02.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 1 -- 172.21.15.86:0/648304599 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8680307d0 con 0x7ff87413f0f0 2023-12-11T20:17:02.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: dump: 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: _finish_auth 0 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:17:02.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 1 -- 172.21.15.86:0/648304599 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff868030930 con 0x7ff87413f0f0 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 1 -- 172.21.15.86:0/648304599 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868030ce0 con 0x7ff87413f0f0 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff862ffd640 10 monclient: dump: 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.173 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 5 monclient: authenticate success, global_id 4341 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 10 monclient: _renew_subs 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff870d99640 4 set_mon_vals no callback set 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff874132d30 con 0x7ff87413f0f0 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 10 monclient: _renew_subs 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8741331f0 con 0x7ff87413f0f0 2023-12-11T20:17:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.173+0000 7ff878d97640 1 librados: init done 2023-12-11T20:17:02.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.174+0000 7ff878d97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.175+0000 7ff878d97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.175+0000 7ff878d97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:02.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.175+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff874137400 con 0x7ff87413f0f0 2023-12-11T20:17:02.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.234+0000 7f4f3effd640 1 -- 172.21.15.86:0/67367539 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4f30015020 con 0x7f4f480a8e00 2023-12-11T20:17:02.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.234+0000 7f4f3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:02.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.234+0000 7f4f3effd640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:68719476741 2023-12-11T20:17:02.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.235+0000 7f34ce52c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-11T20:17:02.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.235+0000 7f34ce52c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-11T20:17:02.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.235+0000 7f34ce52c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.235+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f34c81da440 con 0x7f34c8143330 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 10 librados: watch_flush enter 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 10 librados: watch_flush exit 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 20 client.4316.objecter shutdown clearing up homeless session... 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 10 client.4316.objecter successfully canceled tick 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4f2806c870 msgr2=0x7f4f2806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4f2806c870 0x7f4f2806ed30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005880 tx=0x7f4f38009040 comp rx=0 tx=0).stop 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 10 monclient: shutdown 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f480a8e00 msgr2=0x7f4f48138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f480a8e00 0x7f4f48138cc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000b3e0 tx=0x7f4f3000b8a0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 shutdown_connections 2023-12-11T20:17:02.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f4f2806c870 0x7f4f2806ed30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f480a8e00 0x7f4f48138cc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f481461c0 0x7f4f48140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 --2- 172.21.15.86:0/67367539 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f480a8a20 0x7f4f480b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 >> 172.21.15.86:0/67367539 conn(0x7f4f480b7890 msgr2=0x7f4f481348c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 shutdown_connections 2023-12-11T20:17:02.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 -- 172.21.15.86:0/67367539 wait complete. 2023-12-11T20:17:02.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.237+0000 7f4f4cc4f640 1 librados: shutdown 2023-12-11T20:17:02.250 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.6 2023-12-11T20:17:02.300 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.302+0000 7faf227fc640 1 -- 172.21.15.86:0/1993243145 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faf28007bb0 con 0x7faf2c0a8a20 2023-12-11T20:17:02.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.302+0000 7faf227fc640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7faf1006c820 0x7faf1006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.303+0000 7faf3174b640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7faf1006c820 0x7faf1006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:02.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.303+0000 7faf3174b640 10 monclient: get_auth_request con 0x7faf1006c820 auth_method 0 2023-12-11T20:17:02.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.303+0000 7faf3174b640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7faf1006c820 0x7faf1006ece0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf1c036000 tx=0x7faf1c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.303+0000 7faf227fc640 10 client.4420.objecter ms_handle_connect 0x7faf1006c820 2023-12-11T20:17:02.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.305+0000 7faf227fc640 1 -- 172.21.15.86:0/1993243145 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faf28058e00 con 0x7faf2c0a8a20 2023-12-11T20:17:02.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.305+0000 7faf227fc640 10 client.4420.objecter ms_dispatch 0x7faf2c000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:02.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.305+0000 7faf227fc640 3 client.4420.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:02.304 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.305+0000 7faf227fc640 3 client.4420.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:02.304 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.305+0000 7faf227fc640 20 client.4420.objecter dump_active .. 0 homeless 2023-12-11T20:17:02.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 1 -- 172.21.15.86:0/4120419735 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4011830 con 0x7f54b413f0f0 2023-12-11T20:17:02.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:02.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: dump: 2023-12-11T20:17:02.305 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.305 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.305 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.305 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: _finish_auth 0 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 1 -- 172.21.15.86:0/4120419735 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4011990 con 0x7f54b413f0f0 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 5 monclient: authenticate success, global_id 4423 2023-12-11T20:17:02.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 10 monclient: _renew_subs 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b4145370 con 0x7f54b413f0f0 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 1 -- 172.21.15.86:0/4120419735 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4011d40 con 0x7f54b413f0f0 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54a9ffb640 10 monclient: dump: 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.307 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:02.308 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:02.308 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:02.308 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:02.308 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:02.308 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:02.308 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:02.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54ab7fe640 4 set_mon_vals no callback set 2023-12-11T20:17:02.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:02.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 10 monclient: _renew_subs 2023-12-11T20:17:02.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b41332a0 con 0x7f54b413f0f0 2023-12-11T20:17:02.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.306+0000 7f54bb3f0640 1 librados: init done 2023-12-11T20:17:02.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.307+0000 7f54bb3f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.307+0000 7f54bb3f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.307+0000 7f54bb3f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.307+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b404e840 con 0x7f54b413f0f0 2023-12-11T20:17:02.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.316+0000 7faf227fc640 1 -- 172.21.15.86:0/1993243145 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf28054a40 con 0x7faf2c0a8a20 2023-12-11T20:17:02.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.316+0000 7faf227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.316+0000 7faf227fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:02.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.335+0000 7f814bf83640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:02.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.335+0000 7f814bf83640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:02.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.335+0000 7f814bf83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:02.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.335+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f81441d7160 con 0x7f81440a8a20 2023-12-11T20:17:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.342+0000 7f813a7fc640 1 -- 172.21.15.86:0/690481854 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f81441d7160 con 0x7f81440a8a20 2023-12-11T20:17:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.342+0000 7f813a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.342+0000 7f813a7fc640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:68719476741 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 10 librados: watch_flush enter 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 10 librados: watch_flush exit 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 20 client.4322.objecter shutdown clearing up homeless session... 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 10 client.4322.objecter successfully canceled tick 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f812806c820 msgr2=0x7f812806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f812806c820 0x7f812806ece0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f81440b7410 tx=0x7f8134006bf0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 10 monclient: shutdown 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81440a8a20 msgr2=0x7f8144136ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81440a8a20 0x7f8144136ec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f814000c410 tx=0x7f814000c8d0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.344+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 shutdown_connections 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.345+0000 7f814bf83640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f812806c820 0x7f812806ece0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.345+0000 7f814bf83640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81441461c0 0x7f8144140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.345+0000 7f814bf83640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f814413f0f0 0x7f81441d7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.345+0000 7f814bf83640 1 --2- 172.21.15.86:0/690481854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81440a8a20 0x7f8144136ec0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.345+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 >> 172.21.15.86:0/690481854 conn(0x7f81440b7890 msgr2=0x7f8144145ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.345+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 shutdown_connections 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.345+0000 7f814bf83640 1 -- 172.21.15.86:0/690481854 wait complete. 2023-12-11T20:17:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.345+0000 7f814bf83640 1 librados: shutdown 2023-12-11T20:17:02.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.2 2023-12-11T20:17:02.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.366+0000 7f8aad690640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:02.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.366+0000 7f8aad690640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:02.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.366+0000 7f8aad690640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:02.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.366+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8aa8138d80 con 0x7f8aa81461c0 2023-12-11T20:17:02.448 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.449+0000 7ff862ffd640 1 -- 172.21.15.86:0/648304599 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff86804d450 con 0x7ff87413f0f0 2023-12-11T20:17:02.448 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.450+0000 7ff862ffd640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff85406c850 0x7ff85406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.448 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.450+0000 7ff87259c640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff85406c850 0x7ff85406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:02.448 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.450+0000 7ff87259c640 10 monclient: get_auth_request con 0x7ff85406c850 auth_method 0 2023-12-11T20:17:02.449 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.450+0000 7ff87259c640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff85406c850 0x7ff85406ed10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff864009670 tx=0x7ff864006bf0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.449 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.450+0000 7ff862ffd640 10 client.4341.objecter ms_handle_connect 0x7ff85406c850 2023-12-11T20:17:02.461 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.462+0000 7ff862ffd640 1 -- 172.21.15.86:0/648304599 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff86802d040 con 0x7ff87413f0f0 2023-12-11T20:17:02.461 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.463+0000 7ff862ffd640 10 client.4341.objecter ms_dispatch 0x7ff874000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:02.461 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.463+0000 7ff862ffd640 3 client.4341.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:02.461 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.463+0000 7ff862ffd640 3 client.4341.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:02.461 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.463+0000 7ff862ffd640 20 client.4341.objecter dump_active .. 0 homeless 2023-12-11T20:17:02.461 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.463+0000 7ff862ffd640 1 -- 172.21.15.86:0/648304599 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff868077a00 con 0x7ff87413f0f0 2023-12-11T20:17:02.461 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.463+0000 7ff862ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.462 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.463+0000 7ff862ffd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:02.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.476+0000 7f8a8f7fe640 1 -- 172.21.15.86:0/3346972646 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8aa8138d80 con 0x7f8aa81461c0 2023-12-11T20:17:02.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.476+0000 7f8a8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:02.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.476+0000 7f8a8f7fe640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:55834574851 2023-12-11T20:17:02.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 10 librados: watch_flush enter 2023-12-11T20:17:02.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 10 librados: watch_flush exit 2023-12-11T20:17:02.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 20 client.4335.objecter shutdown clearing up homeless session... 2023-12-11T20:17:02.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 10 client.4335.objecter successfully canceled tick 2023-12-11T20:17:02.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8a9406c870 msgr2=0x7f8a9406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8a9406c870 0x7f8a9406ed30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a98000f60 tx=0x7f8a980079d0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 10 monclient: shutdown 2023-12-11T20:17:02.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa81461c0 msgr2=0x7f8aa8135650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa81461c0 0x7f8aa8135650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009e50 tx=0x7f8aa0004a40 comp rx=0 tx=0).stop 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 shutdown_connections 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8a9406c870 0x7f8a9406ed30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa81461c0 0x7f8aa8135650 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8aa813f0f0 0x7f8aa8139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 --2- 172.21.15.86:0/3346972646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa80a8a20 0x7f8aa8135050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 >> 172.21.15.86:0/3346972646 conn(0x7f8aa80b7890 msgr2=0x7f8aa80b7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 shutdown_connections 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 -- 172.21.15.86:0/3346972646 wait complete. 2023-12-11T20:17:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.478+0000 7f8aad690640 1 librados: shutdown 2023-12-11T20:17:02.488 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.5 2023-12-11T20:17:02.498 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.500+0000 7faf341b0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:02.499 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.500+0000 7faf341b0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:02.499 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.500+0000 7faf341b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.499 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.500+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7faf2c04e840 con 0x7faf2c0a8a20 2023-12-11T20:17:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.560+0000 7f54a9ffb640 1 -- 172.21.15.86:0/4120419735 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f54a401a400 con 0x7f54b413f0f0 2023-12-11T20:17:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.560+0000 7f54a9ffb640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f549806c820 0x7f549806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.560+0000 7f54b918c640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f549806c820 0x7f549806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.560+0000 7f54b918c640 10 monclient: get_auth_request con 0x7f549806c820 auth_method 0 2023-12-11T20:17:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.561+0000 7f54a9ffb640 10 client.4423.objecter ms_handle_connect 0x7f549806c820 2023-12-11T20:17:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.561+0000 7f54b918c640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f549806c820 0x7f549806ece0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f54b0001560 tx=0x7f54b0007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.564+0000 7f54a9ffb640 1 -- 172.21.15.86:0/4120419735 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f54a4059190 con 0x7f54b413f0f0 2023-12-11T20:17:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.564+0000 7f54a9ffb640 10 client.4423.objecter ms_dispatch 0x7f54b4000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.564+0000 7f54a9ffb640 3 client.4423.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.564+0000 7f54a9ffb640 3 client.4423.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.564+0000 7f54a9ffb640 20 client.4423.objecter dump_active .. 0 homeless 2023-12-11T20:17:02.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.576+0000 7f54a9ffb640 1 -- 172.21.15.86:0/4120419735 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54a4054dd0 con 0x7f54b413f0f0 2023-12-11T20:17:02.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.576+0000 7f54a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:02.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.576+0000 7f54a9ffb640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:02.579 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.581+0000 7faf227fc640 1 -- 172.21.15.86:0/1993243145 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faf2801e070 con 0x7faf2c0a8a20 2023-12-11T20:17:02.580 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.581+0000 7faf227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:02.580 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.581+0000 7faf227fc640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:17:02.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 10 librados: watch_flush enter 2023-12-11T20:17:02.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 10 librados: watch_flush exit 2023-12-11T20:17:02.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 20 client.4420.objecter shutdown clearing up homeless session... 2023-12-11T20:17:02.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 10 client.4420.objecter successfully canceled tick 2023-12-11T20:17:02.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7faf1006c820 msgr2=0x7faf1006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7faf1006c820 0x7faf1006ece0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf1c036000 tx=0x7faf1c030000 comp rx=0 tx=0).stop 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 10 monclient: shutdown 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c0a8a20 msgr2=0x7faf2c138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.583+0000 7faf341b0640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c0a8a20 0x7faf2c138190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faf2800c7f0 tx=0x7faf2800ccb0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 shutdown_connections 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7faf1006c820 0x7faf1006ece0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf2c145ab0 0x7faf2c1389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf2c13f0f0 0x7faf2c144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 --2- 172.21.15.86:0/1993243145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf2c0a8a20 0x7faf2c138190 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 >> 172.21.15.86:0/1993243145 conn(0x7faf2c0b7890 msgr2=0x7faf2c0b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 shutdown_connections 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 -- 172.21.15.86:0/1993243145 wait complete. 2023-12-11T20:17:02.583 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.584+0000 7faf341b0640 1 librados: shutdown 2023-12-11T20:17:02.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.0 2023-12-11T20:17:02.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.638+0000 7f34c4ff9640 1 -- 172.21.15.86:0/4220697354 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f34c81da440 con 0x7f34c8143330 2023-12-11T20:17:02.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.638+0000 7f34c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-11T20:17:02.637 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.638+0000 7f34c4ff9640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:55834574854 2023-12-11T20:17:02.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.640+0000 7f34ce52c640 10 librados: watch_flush enter 2023-12-11T20:17:02.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.640+0000 7f34ce52c640 10 librados: watch_flush exit 2023-12-11T20:17:02.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.640+0000 7f34ce52c640 20 client.4411.objecter shutdown clearing up homeless session... 2023-12-11T20:17:02.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.640+0000 7f34ce52c640 10 client.4411.objecter successfully canceled tick 2023-12-11T20:17:02.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f34b406c820 msgr2=0x7f34b406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f34b406c820 0x7f34b406ece0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f34b80014e0 tx=0x7f34b8007990 comp rx=0 tx=0).stop 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 10 monclient: shutdown 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34c8143330 msgr2=0x7f34c8141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34c8143330 0x7f34c8141370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c8f0 tx=0x7f34bc00cdb0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 shutdown_connections 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f34b406c820 0x7f34b406ece0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34c81dab00 0x7f34c81daf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34c8143330 0x7f34c8141370 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 --2- 172.21.15.86:0/4220697354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0004d50 0x7f34c8140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 >> 172.21.15.86:0/4220697354 conn(0x7f34c80b7890 msgr2=0x7f34c80b5690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 shutdown_connections 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 -- 172.21.15.86:0/4220697354 wait complete. 2023-12-11T20:17:02.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.641+0000 7f34ce52c640 1 librados: shutdown 2023-12-11T20:17:02.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2023-12-11T20:17:02.651 DEBUG:teuthology.parallel:result is None 2023-12-11T20:17:02.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.721+0000 7ff878d97640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-11T20:17:02.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.721+0000 7ff878d97640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-11T20:17:02.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.721+0000 7ff878d97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:02.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.721+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff87404e840 con 0x7ff87413f0f0 2023-12-11T20:17:02.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.725+0000 7ff862ffd640 1 -- 172.21.15.86:0/648304599 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff86807b020 con 0x7ff87413f0f0 2023-12-11T20:17:02.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.725+0000 7ff862ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-11T20:17:02.723 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.725+0000 7ff862ffd640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:68719476743 2023-12-11T20:17:02.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 10 librados: watch_flush enter 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 10 librados: watch_flush exit 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 20 client.4341.objecter shutdown clearing up homeless session... 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 10 client.4341.objecter successfully canceled tick 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff85406c850 msgr2=0x7ff85406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff85406c850 0x7ff85406ed10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff864009670 tx=0x7ff864006bf0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 10 monclient: shutdown 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff87413f0f0 msgr2=0x7ff874136ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff87413f0f0 0x7ff874136ec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff8680090b0 tx=0x7ff8680046c0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 shutdown_connections 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7ff85406c850 0x7ff85406ed10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff874145ab0 0x7ff874137bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff87413f0f0 0x7ff874136ec0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 --2- 172.21.15.86:0/648304599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8740a8a20 0x7ff8740b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 >> 172.21.15.86:0/648304599 conn(0x7ff8740b7890 msgr2=0x7ff8740b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 shutdown_connections 2023-12-11T20:17:02.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 -- 172.21.15.86:0/648304599 wait complete. 2023-12-11T20:17:02.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.727+0000 7ff878d97640 1 librados: shutdown 2023-12-11T20:17:02.736 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.3 2023-12-11T20:17:02.736 DEBUG:teuthology.parallel:result is None 2023-12-11T20:17:02.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.823+0000 7f54bb3f0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-11T20:17:02.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.823+0000 7f54bb3f0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-11T20:17:02.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.823+0000 7f54bb3f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:02.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.823+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f54b41daf70 con 0x7f54b413f0f0 2023-12-11T20:17:02.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.827+0000 7f54a9ffb640 1 -- 172.21.15.86:0/4120419735 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f54a4015020 con 0x7f54b413f0f0 2023-12-11T20:17:02.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.827+0000 7f54a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-11T20:17:02.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.827+0000 7f54a9ffb640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:17:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 10 librados: watch_flush enter 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 10 librados: watch_flush exit 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 20 client.4423.objecter shutdown clearing up homeless session... 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 10 client.4423.objecter successfully canceled tick 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f549806c820 msgr2=0x7f549806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f549806c820 0x7f549806ece0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f54b0001560 tx=0x7f54b0007990 comp rx=0 tx=0).stop 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 10 monclient: shutdown 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 msgr2=0x7f54b41da820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b41da820 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b410 tx=0x7f54a400b8d0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 shutdown_connections 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f549806c820 0x7f549806ece0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b41461c0 0x7f54b4132b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54b413f0f0 0x7f54b41da820 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 --2- 172.21.15.86:0/4120419735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54b40a8a20 0x7f54b40b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:02.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 >> 172.21.15.86:0/4120419735 conn(0x7f54b40b7890 msgr2=0x7f54b40b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:02.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 shutdown_connections 2023-12-11T20:17:02.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 -- 172.21.15.86:0/4120419735 wait complete. 2023-12-11T20:17:02.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:02.829+0000 7f54bb3f0640 1 librados: shutdown 2023-12-11T20:17:02.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.1 2023-12-11T20:17:02.840 DEBUG:teuthology.parallel:result is None 2023-12-11T20:17:03.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-11T20:17:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.245+0000 7fb20b89b640 10 monclient: get_monmap_and_config 2023-12-11T20:17:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.245+0000 7fb20b89b640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.245+0000 7fb20b89b640 10 monclient: monmap: 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:03.246094+0000 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:03.246094+0000 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:03.244 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.246+0000 7fb20b89b640 1 Processor -- start 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.246+0000 7fb20b89b640 1 -- start start 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.246+0000 7fb20b89b640 10 monclient: init 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.246+0000 7fb20b89b640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.246+0000 7fb20b89b640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.246+0000 7fb20b89b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb2040a8a20 0x7fb20413eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.246+0000 7fb20b89b640 10 monclient(hunting): picked mon.noname-a con 0x7fb2040a8a20 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fb2040a8e00 0x7fb20413f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 10 monclient(hunting): picked mon.noname-c con 0x7fb2040a8e00 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2041423b0 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2040b21f0 con 0x7fb2041423b0 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2040a9cc0 con 0x7fb2040a8a20 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb204142f50 con 0x7fb2040a8e00 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb20b89b640 10 monclient(hunting): authenticate will time out at 1509.576186s 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb209637640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb2040a8a20 0x7fb20413eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.247+0000 7fb208e36640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fb2040a8e00 0x7fb20413f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb209637640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb2040a8a20 0x7fb20413eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35122/0 (socket says 172.21.15.86:35122) 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb209637640 1 -- 172.21.15.86:0/773160700 learned_addr learned my addr 172.21.15.86:0/773160700 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb209637640 10 monclient(hunting): get_auth_request con 0x7fb2040a8a20 auth_method 0 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb209637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb209637640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb209637640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb208e36640 10 monclient(hunting): get_auth_request con 0x7fb2040a8e00 auth_method 0 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb208e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb208e36640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb208e36640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb208e36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb208e36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.248+0000 7fb208e36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.249+0000 7fb209637640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.249+0000 7fb209637640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.249+0000 7fb209637640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 10 monclient(hunting): handle_auth_done global_id 4426 payload 274 2023-12-11T20:17:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 1 -- 172.21.15.86:0/773160700 >> v2:172.21.15.86:3300/0 conn(0x7fb2040a8a20 msgr2=0x7fb20413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 1 --2- 172.21.15.86:0/773160700 >> v2:172.21.15.86:3300/0 conn(0x7fb2040a8a20 0x7fb20413eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 1 -- 172.21.15.86:0/773160700 >> v1:172.21.15.86:6789/0 conn(0x7fb2041423b0 legacy=0x7fb204142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 1 monclient: found mon.noname-c 2023-12-11T20:17:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.250+0000 7fb208e36640 1 -- 172.21.15.86:0/773160700 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20413f930 con 0x7fb2040a8e00 2023-12-11T20:17:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.252+0000 7fb208e36640 1 --2- 172.21.15.86:0/773160700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413f3f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb2040b5370 tx=0x7fb1f402b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c19b50b171333d3e server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 1 -- 172.21.15.86:0/773160700 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4031710 con 0x7fb2040a8e00 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: dump: 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 1 -- 172.21.15.86:0/773160700 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f402f050 con 0x7fb2040a8e00 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 1 -- 172.21.15.86:0/773160700 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4031a90 con 0x7fb2040a8e00 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb20b89b640 5 monclient: authenticate success, global_id 4426 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb20b89b640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb20b89b640 4 set_mon_vals no callback set 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb1fbfff640 10 monclient: dump: 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.256 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb20b89b640 10 monclient: shutdown 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb20b89b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb20b89b640 1 -- 172.21.15.86:0/773160700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 msgr2=0x7fb20413f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.256+0000 7fb20b89b640 1 --2- 172.21.15.86:0/773160700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413f3f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb2040b5370 tx=0x7fb1f402b7b0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.257+0000 7fb20b89b640 1 -- 172.21.15.86:0/773160700 shutdown_connections 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.257+0000 7fb20b89b640 1 --2- 172.21.15.86:0/773160700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413f3f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.257+0000 7fb20b89b640 1 --2- 172.21.15.86:0/773160700 >> v2:172.21.15.86:3300/0 conn(0x7fb2040a8a20 0x7fb20413eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.257+0000 7fb20b89b640 1 -- 172.21.15.86:0/773160700 >> 172.21.15.86:0/773160700 conn(0x7fb2040b7870 msgr2=0x7fb2040b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.257+0000 7fb20b89b640 1 -- 172.21.15.86:0/773160700 shutdown_connections 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.257+0000 7fb20b89b640 1 -- 172.21.15.86:0/773160700 wait complete. 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.257+0000 7fb20b89b640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.257+0000 7fb20b89b640 10 monclient: monmap: 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 librados: starting msgr at 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 librados: starting objecter 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 Processor -- start 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 -- start start 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 librados: setting wanted keys 2023-12-11T20:17:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 librados: calling monclient init 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient: init 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2040a8a20 0x7fb204146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient(hunting): picked mon.noname-c con 0x7fb2040a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient(hunting): picked mon.noname-b con 0x7fb2040a8e00 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2041423b0 0x7fb2041d7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient(hunting): picked mon.noname-a con 0x7fb2041423b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb204144c50 con 0x7fb2041423b0 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2041d7060 con 0x7fb2040a8e00 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2041d7340 con 0x7fb2040a8a20 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.258+0000 7fb20b89b640 10 monclient(hunting): authenticate will time out at 1509.587936s 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209e38640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2041423b0 0x7fb2041d7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb208e36640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209637640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2040a8a20 0x7fb204146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb208e36640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52548/0 (socket says 172.21.15.86:52548) 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb208e36640 1 -- 172.21.15.86:0/3264988789 learned_addr learned my addr 172.21.15.86:0/3264988789 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb208e36640 10 monclient(hunting): get_auth_request con 0x7fb2040a8e00 auth_method 0 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb208e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb208e36640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb208e36640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209e38640 10 monclient(hunting): get_auth_request con 0x7fb2041423b0 auth_method 0 2023-12-11T20:17:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209e38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209e38640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209e38640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209637640 10 monclient(hunting): get_auth_request con 0x7fb2040a8a20 auth_method 0 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209637640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.259+0000 7fb209637640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb208e36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb208e36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb208e36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb209e38640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb209e38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb209e38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb209637640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb209637640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.260+0000 7fb209637640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2023-12-11T20:17:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 1 -- 172.21.15.86:0/3264988789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2040a8a20 msgr2=0x7fb204146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2040a8a20 0x7fb204146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 1 -- 172.21.15.86:0/3264988789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2041423b0 msgr2=0x7fb2041d7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2041423b0 0x7fb2041d7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 1 monclient: found mon.noname-b 2023-12-11T20:17:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.262+0000 7fb208e36640 1 -- 172.21.15.86:0/3264988789 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f4009400 con 0x7fb2040a8e00 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.264+0000 7fb208e36640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413aeb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4000c00 tx=0x7fb1f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.264+0000 7fb1f9ffb640 10 client.?.objecter ms_handle_connect 0x7fb2040a8e00 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.264+0000 7fb1f9ffb640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 1 -- 172.21.15.86:0/3264988789 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4033890 con 0x7fb2040a8e00 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: dump: 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 5 monclient: authenticate success, global_id 4429 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 1 -- 172.21.15.86:0/3264988789 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f40339f0 con 0x7fb2040a8e00 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 1 -- 172.21.15.86:0/3264988789 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4033da0 con 0x7fb2040a8e00 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1f9ffb640 10 monclient: dump: 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb1fb7fe640 4 set_mon_vals no callback set 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 10 monclient: _renew_subs 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2041d7da0 con 0x7fb2040a8e00 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 10 monclient: _renew_subs 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb204138df0 con 0x7fb2040a8e00 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.268+0000 7fb20b89b640 1 librados: init done 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.269+0000 7fb20b89b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.269+0000 7fb20b89b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.269+0000 7fb20b89b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.269+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2040a9640 con 0x7fb2040a8e00 2023-12-11T20:17:03.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.353+0000 7f37250a2640 10 monclient: get_monmap_and_config 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.353+0000 7f37250a2640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.353+0000 7f37250a2640 10 monclient: monmap: 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:03.353805+0000 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:03.353805+0000 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:03.352 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:03.353 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.353+0000 7f37250a2640 1 Processor -- start 2023-12-11T20:17:03.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.353+0000 7f37250a2640 1 -- start start 2023-12-11T20:17:03.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.353+0000 7f37250a2640 10 monclient: init 2023-12-11T20:17:03.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-12-11T20:17:03.354 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37200a8a20 0x7f37200a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 10 monclient(hunting): picked mon.noname-a con 0x7f37200a8a20 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3720145ab0 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f372013f800 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3720145620 con 0x7f372013f800 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37200aace0 con 0x7f3720145ab0 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37200aafc0 con 0x7f37200a8a20 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.354+0000 7f37250a2640 10 monclient(hunting): authenticate will time out at 1509.683668s 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.355+0000 7f371ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37200a8a20 0x7f37200a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.355+0000 7f371ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37200a8a20 0x7f37200a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35134/0 (socket says 172.21.15.86:35134) 2023-12-11T20:17:03.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.355+0000 7f371ed9d640 1 -- 172.21.15.86:0/1307908709 learned_addr learned my addr 172.21.15.86:0/1307908709 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.355+0000 7f371ed9d640 10 monclient(hunting): get_auth_request con 0x7f37200a8a20 auth_method 0 2023-12-11T20:17:03.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.355+0000 7f371ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.355+0000 7f371ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.355+0000 7f371ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.356+0000 7f371ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.356+0000 7f371ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.356+0000 7f371ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-11T20:17:03.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.358+0000 7f371ed9d640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2023-12-11T20:17:03.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.358+0000 7f371ed9d640 1 -- 172.21.15.86:0/1307908709 >> v1:172.21.15.133:6789/0 conn(0x7f3720145ab0 legacy=0x7f372013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.358+0000 7f371ed9d640 1 -- 172.21.15.86:0/1307908709 >> v1:172.21.15.86:6789/0 conn(0x7f372013f800 legacy=0x7f372014c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.358+0000 7f371ed9d640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.358+0000 7f371ed9d640 1 monclient: found mon.noname-a 2023-12-11T20:17:03.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.358+0000 7f371ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.358+0000 7f371ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:03.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.358+0000 7f371ed9d640 1 -- 172.21.15.86:0/1307908709 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37200b2210 con 0x7f37200a8a20 2023-12-11T20:17:03.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.361+0000 7f371ed9d640 1 --2- 172.21.15.86:0/1307908709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f37200a8e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3710009e50 tx=0x7f371002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3f61bbd9563571 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 1 -- 172.21.15.86:0/1307908709 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371002cb20 con 0x7f37200a8a20 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: dump: 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 1 -- 172.21.15.86:0/1307908709 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371002cc80 con 0x7f37200a8a20 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 1 -- 172.21.15.86:0/1307908709 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37100343f0 con 0x7f37200a8a20 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f37250a2640 5 monclient: authenticate success, global_id 4353 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f37250a2640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f37250a2640 4 set_mon_vals no callback set 2023-12-11T20:17:03.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f371dd9b640 10 monclient: dump: 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.367 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.368 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f37250a2640 10 monclient: shutdown 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f37250a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f37250a2640 1 -- 172.21.15.86:0/1307908709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 msgr2=0x7f37200a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.367+0000 7f37250a2640 1 --2- 172.21.15.86:0/1307908709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f37200a8e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3710009e50 tx=0x7f371002bf80 comp rx=0 tx=0).stop 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.368+0000 7f37250a2640 1 -- 172.21.15.86:0/1307908709 shutdown_connections 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.368+0000 7f37250a2640 1 --2- 172.21.15.86:0/1307908709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f37200a8e00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.368+0000 7f37250a2640 1 -- 172.21.15.86:0/1307908709 >> 172.21.15.86:0/1307908709 conn(0x7f37200b7890 msgr2=0x7f37200b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.368+0000 7f37250a2640 1 -- 172.21.15.86:0/1307908709 shutdown_connections 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.368+0000 7f37250a2640 1 -- 172.21.15.86:0/1307908709 wait complete. 2023-12-11T20:17:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.368+0000 7f37250a2640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.368+0000 7f37250a2640 10 monclient: monmap: 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.369+0000 7f37250a2640 1 librados: starting msgr at 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.369+0000 7f37250a2640 1 librados: starting objecter 2023-12-11T20:17:03.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.369+0000 7f37250a2640 1 Processor -- start 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.369+0000 7f37250a2640 1 -- start start 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.369+0000 7f37250a2640 1 librados: setting wanted keys 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.369+0000 7f37250a2640 1 librados: calling monclient init 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.369+0000 7f37250a2640 10 monclient: init 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.369+0000 7f37250a2640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f3720142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient(hunting): picked mon.noname-a con 0x7f37200a8a20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f372013f800 0x7f3720143860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient(hunting): picked mon.noname-b con 0x7f372013f800 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720145ab0 0x7f37201d72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient(hunting): picked mon.noname-c con 0x7f3720145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3720144b50 con 0x7f37200a8a20 2023-12-11T20:17:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3720144d50 con 0x7f372013f800 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3720145030 con 0x7f3720145ab0 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f37250a2640 10 monclient(hunting): authenticate will time out at 1509.699204s 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f371f59e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720145ab0 0x7f37201d72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.370+0000 7f371ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f3720142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f3720142bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35142/0 (socket says 172.21.15.86:35142) 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 1 -- 172.21.15.86:0/2574444183 learned_addr learned my addr 172.21.15.86:0/2574444183 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 10 monclient(hunting): get_auth_request con 0x7f37200a8a20 auth_method 0 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371f59e640 10 monclient(hunting): get_auth_request con 0x7f3720145ab0 auth_method 0 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371f59e640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371f59e640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.371+0000 7f371f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 1 -- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f372013f800 msgr2=0x7f3720143860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f372013f800 0x7f3720143860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 1 -- 172.21.15.86:0/2574444183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 msgr2=0x7f3720142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f3720142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 1 monclient: found mon.noname-c 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371f59e640 1 -- 172.21.15.86:0/2574444183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710009b30 con 0x7f3720145ab0 2023-12-11T20:17:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.374+0000 7f371ed9d640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f3720142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:17:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.377+0000 7f371f59e640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720145ab0 0x7f37201d72e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f371800c3c0 tx=0x7f371800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.377+0000 7f37077fe640 10 client.?.objecter ms_handle_connect 0x7f3720145ab0 2023-12-11T20:17:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.377+0000 7f37077fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 1 -- 172.21.15.86:0/2574444183 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37180052d0 con 0x7f3720145ab0 2023-12-11T20:17:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: dump: 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 1 -- 172.21.15.86:0/2574444183 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3718005430 con 0x7f3720145ab0 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 5 monclient: authenticate success, global_id 4328 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 10 monclient: _renew_subs 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3720143ef0 con 0x7f3720145ab0 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 1 -- 172.21.15.86:0/2574444183 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718007260 con 0x7f3720145ab0 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37077fe640 10 monclient: dump: 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f371d59a640 4 set_mon_vals no callback set 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 10 monclient: _renew_subs 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37201d7a00 con 0x7f3720145ab0 2023-12-11T20:17:03.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.380+0000 7f37250a2640 1 librados: init done 2023-12-11T20:17:03.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.381+0000 7f37250a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.381+0000 7f37250a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.381+0000 7f37250a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.381+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37201d7000 con 0x7f3720145ab0 2023-12-11T20:17:03.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.402+0000 7f37077fe640 1 -- 172.21.15.86:0/2574444183 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f37180073c0 con 0x7f3720145ab0 2023-12-11T20:17:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.403+0000 7f37077fe640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f370c06c870 0x7f370c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.403+0000 7f371ed9d640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f370c06c870 0x7f370c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.403+0000 7f371ed9d640 10 monclient: get_auth_request con 0x7f370c06c870 auth_method 0 2023-12-11T20:17:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.404+0000 7f371ed9d640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f370c06c870 0x7f370c06ed30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f371002bfc0 tx=0x7f3710030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.404+0000 7f37077fe640 10 client.4328.objecter ms_handle_connect 0x7f370c06c870 2023-12-11T20:17:03.462 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.464+0000 7fb682590640 10 monclient: get_monmap_and_config 2023-12-11T20:17:03.462 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.464+0000 7fb682590640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.462 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.464+0000 7fb682590640 10 monclient: monmap: 2023-12-11T20:17:03.462 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.462 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:03.464947+0000 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:03.464947+0000 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:03.463 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 1 Processor -- start 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 1 -- start start 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient: init 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb67c0a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fb67c1461c0 0x7fb67c0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient(hunting): picked mon.noname-c con 0x7fb67c1461c0 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb67c13f0f0 0x7fb67c13f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.465 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient(hunting): picked mon.noname-a con 0x7fb67c13f0f0 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c145620 con 0x7fb67c0a8a20 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67c0aace0 con 0x7fb67c13f0f0 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67c0aafc0 con 0x7fb67c1461c0 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.465+0000 7fb682590640 10 monclient(hunting): authenticate will time out at 1509.794970s 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.466+0000 7fb680b2d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb67c13f0f0 0x7fb67c13f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.466+0000 7fb680b2d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb67c13f0f0 0x7fb67c13f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35158/0 (socket says 172.21.15.86:35158) 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.466+0000 7fb680b2d640 1 -- 172.21.15.86:0/2881278423 learned_addr learned my addr 172.21.15.86:0/2881278423 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.466+0000 7fb680b2d640 10 monclient(hunting): get_auth_request con 0x7fb67c13f0f0 auth_method 0 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.466+0000 7fb680b2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.466 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.466+0000 7fb680b2d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.467 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.466+0000 7fb680b2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.467 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.467+0000 7fb680b2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.467 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.467+0000 7fb680b2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.467 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.467+0000 7fb680b2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.467 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2023-12-11T20:17:03.467 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 1 -- 172.21.15.86:0/2881278423 >> v2:172.21.15.133:3300/0 conn(0x7fb67c1461c0 msgr2=0x7fb67c0a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 1 --2- 172.21.15.86:0/2881278423 >> v2:172.21.15.133:3300/0 conn(0x7fb67c1461c0 0x7fb67c0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 1 -- 172.21.15.86:0/2881278423 >> v1:172.21.15.133:6789/0 conn(0x7fb67c0a8a20 legacy=0x7fb67c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 1 monclient: found mon.noname-a 2023-12-11T20:17:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.468+0000 7fb680b2d640 1 -- 172.21.15.86:0/2881278423 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c13fa90 con 0x7fb67c13f0f0 2023-12-11T20:17:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.471+0000 7fb680b2d640 1 --2- 172.21.15.86:0/2881278423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c13f0f0 0x7fb67c13f550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0a9a50 tx=0x7fb67402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a20a328e1087ed server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.471 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb67affd640 1 -- 172.21.15.86:0/2881278423 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67402ca80 con 0x7fb67c13f0f0 2023-12-11T20:17:03.471 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.471 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb67affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:03.471 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb67affd640 10 monclient: dump: 2023-12-11T20:17:03.471 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.471 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.471 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.471 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb67affd640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb67affd640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb67affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb682590640 5 monclient: authenticate success, global_id 4362 2023-12-11T20:17:03.472 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.473+0000 7fb682590640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.474+0000 7fb67affd640 1 -- 172.21.15.86:0/2881278423 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67402cbe0 con 0x7fb67c13f0f0 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.474+0000 7fb67affd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.474+0000 7fb67affd640 1 -- 172.21.15.86:0/2881278423 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740343f0 con 0x7fb67c13f0f0 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.474+0000 7fb67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.474+0000 7fb67affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.474+0000 7fb67affd640 10 monclient: dump: 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.474 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.474+0000 7fb682590640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.474+0000 7fb682590640 4 set_mon_vals no callback set 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 10 monclient: shutdown 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 1 -- 172.21.15.86:0/2881278423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c13f0f0 msgr2=0x7fb67c13f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 1 --2- 172.21.15.86:0/2881278423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c13f0f0 0x7fb67c13f550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0a9a50 tx=0x7fb67402bee0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 1 -- 172.21.15.86:0/2881278423 shutdown_connections 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 1 --2- 172.21.15.86:0/2881278423 >> v2:172.21.15.133:3300/0 conn(0x7fb67c1461c0 0x7fb67c0a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 1 --2- 172.21.15.86:0/2881278423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c13f0f0 0x7fb67c13f550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 1 -- 172.21.15.86:0/2881278423 >> 172.21.15.86:0/2881278423 conn(0x7fb67c0b7890 msgr2=0x7fb67c0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 1 -- 172.21.15.86:0/2881278423 shutdown_connections 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.475+0000 7fb682590640 1 -- 172.21.15.86:0/2881278423 wait complete. 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 10 monclient: monmap: 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 1 librados: starting msgr at 2023-12-11T20:17:03.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 1 librados: starting objecter 2023-12-11T20:17:03.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 1 Processor -- start 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 1 -- start start 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 1 librados: setting wanted keys 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 1 librados: calling monclient init 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.476+0000 7fb682590640 10 monclient: init 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb67c0a8a20 0x7fb67c1441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient(hunting): picked mon.noname-b con 0x7fb67c0a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb67c13f0f0 0x7fb67c1da880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient(hunting): picked mon.noname-c con 0x7fb67c13f0f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c1461c0 0x7fb67c1dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient(hunting): picked mon.noname-a con 0x7fb67c1461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c145320 con 0x7fb67c1461c0 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c144e90 con 0x7fb67c0a8a20 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c137dc0 con 0x7fb67c13f0f0 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb682590640 10 monclient(hunting): authenticate will time out at 1509.806518s 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.477+0000 7fb680b2d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c1461c0 0x7fb67c1dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c1461c0 0x7fb67c1dadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35170/0 (socket says 172.21.15.86:35170) 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 1 -- 172.21.15.86:0/3887873985 learned_addr learned my addr 172.21.15.86:0/3887873985 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 10 monclient(hunting): get_auth_request con 0x7fb67c1461c0 auth_method 0 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.480 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.481 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.481 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.481 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.478+0000 7fb680b2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.481 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2023-12-11T20:17:03.481 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 1 -- 172.21.15.86:0/3887873985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb67c13f0f0 msgr2=0x7fb67c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.481 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb67c13f0f0 0x7fb67c1da880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 1 -- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb67c0a8a20 msgr2=0x7fb67c1441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb67c0a8a20 0x7fb67c1441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 1 monclient: found mon.noname-a 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.481+0000 7fb680b2d640 1 -- 172.21.15.86:0/3887873985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb674009b30 con 0x7fb67c1461c0 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.483+0000 7fb680b2d640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c1461c0 0x7fb67c1dadc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb674036040 tx=0x7fb674004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.483+0000 7fb678ff9640 10 client.?.objecter ms_handle_connect 0x7fb67c1461c0 2023-12-11T20:17:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.483+0000 7fb678ff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:03.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 1 -- 172.21.15.86:0/3887873985 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67403e450 con 0x7fb67c1461c0 2023-12-11T20:17:03.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:03.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: dump: 2023-12-11T20:17:03.485 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.485 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.485 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 1 -- 172.21.15.86:0/3887873985 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67403e5b0 con 0x7fb67c1461c0 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb682590640 5 monclient: authenticate success, global_id 4365 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb682590640 10 monclient: _renew_subs 2023-12-11T20:17:03.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 1 -- 172.21.15.86:0/3887873985 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67403e960 con 0x7fb67c1461c0 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb682590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c1380a0 con 0x7fb67c1461c0 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb678ff9640 10 monclient: dump: 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.487 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.486+0000 7fb67a7fc640 4 set_mon_vals no callback set 2023-12-11T20:17:03.488 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.487+0000 7fb682590640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:03.488 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.487+0000 7fb682590640 10 monclient: _renew_subs 2023-12-11T20:17:03.488 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.487+0000 7fb682590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:03.488 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.487+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c136fe0 con 0x7fb67c1461c0 2023-12-11T20:17:03.488 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.487+0000 7fb682590640 1 librados: init done 2023-12-11T20:17:03.488 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.488+0000 7fb682590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.488 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.488+0000 7fb682590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.488 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.488+0000 7fb682590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:03.489 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.488+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c04e840 con 0x7fb67c1461c0 2023-12-11T20:17:03.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-11T20:17:03.506 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.508+0000 7fb678ff9640 1 -- 172.21.15.86:0/3887873985 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb67402cd50 con 0x7fb67c1461c0 2023-12-11T20:17:03.507 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.509+0000 7fb678ff9640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb66806c820 0x7fb66806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.508 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.509+0000 7fb67bfff640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb66806c820 0x7fb66806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.508 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.509+0000 7fb67bfff640 10 monclient: get_auth_request con 0x7fb66806c820 auth_method 0 2023-12-11T20:17:03.508 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.510+0000 7fb1f9ffb640 1 -- 172.21.15.86:0/3264988789 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb1f403a040 con 0x7fb2040a8e00 2023-12-11T20:17:03.509 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.510+0000 7fb678ff9640 10 client.4365.objecter ms_handle_connect 0x7fb66806c820 2023-12-11T20:17:03.509 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.510+0000 7fb67bfff640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb66806c820 0x7fb66806ece0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb66c001560 tx=0x7fb66c007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.509 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.510+0000 7fb1f9ffb640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb1e806c870 0x7fb1e806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.509 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.510+0000 7fb209637640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb1e806c870 0x7fb1e806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.510+0000 7fb209637640 10 monclient: get_auth_request con 0x7fb1e806c870 auth_method 0 2023-12-11T20:17:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.511+0000 7fb1f9ffb640 10 client.4429.objecter ms_handle_connect 0x7fb1e806c870 2023-12-11T20:17:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.511+0000 7fb209637640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb1e806c870 0x7fb1e806ed30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb200009bd0 tx=0x7fb200009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.512+0000 7fb1f9ffb640 1 -- 172.21.15.86:0/3264988789 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb1f4079dc0 con 0x7fb2040a8e00 2023-12-11T20:17:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.512+0000 7fb1f9ffb640 10 client.4429.objecter ms_dispatch 0x7fb204000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.512+0000 7fb1f9ffb640 3 client.4429.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.512+0000 7fb1f9ffb640 3 client.4429.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.512+0000 7fb1f9ffb640 20 client.4429.objecter dump_active .. 0 homeless 2023-12-11T20:17:03.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.522+0000 7fb1f9ffb640 1 -- 172.21.15.86:0/3264988789 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1f4076100 con 0x7fb2040a8e00 2023-12-11T20:17:03.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.522+0000 7fb1f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.522+0000 7fb1f9ffb640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.593+0000 7f6a7ee89640 10 monclient: get_monmap_and_config 2023-12-11T20:17:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.593+0000 7f6a7ee89640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.593+0000 7f6a7ee89640 10 monclient: monmap: 2023-12-11T20:17:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:03.593834+0000 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:03.593834+0000 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:03.592 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 1 Processor -- start 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 1 -- start start 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient: init 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a780a8a20 0x7f6a781459f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient(hunting): picked mon.noname-a con 0x7f6a780a8a20 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a780a9c80 0x7f6a78145f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient(hunting): picked mon.noname-b con 0x7f6a780a9c80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a780a8e00 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a780aac70 con 0x7f6a780a8e00 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a780aaf50 con 0x7f6a780a8a20 2023-12-11T20:17:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a781454d0 con 0x7f6a780a9c80 2023-12-11T20:17:03.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.594+0000 7f6a7ee89640 10 monclient(hunting): authenticate will time out at 1509.924026s 2023-12-11T20:17:03.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.595+0000 7f6a7d426640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f6a780a8e00 0x7f6a78141ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:60034/0 (socket says 172.21.15.86:60034) 2023-12-11T20:17:03.595 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.595+0000 7f6a7d426640 1 -- 172.21.15.86:0/1485518824 learned_addr learned my addr 172.21.15.86:0/1485518824 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:17:03.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.598+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591936257 0 0) 0x7f6a780aac70 con 0x7f6a780a8e00 2023-12-11T20:17:03.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.598+0000 7f6a777fe640 10 monclient(hunting): _negotiate 2023-12-11T20:17:03.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.598+0000 7f6a777fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.598+0000 7f6a777fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.598+0000 7f6a777fe640 10 monclient(hunting): my global_id is 4432 2023-12-11T20:17:03.596 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.598+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a68003520 con 0x7f6a780a8e00 2023-12-11T20:17:03.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-11T20:17:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 134329014 0 0) 0x7f6a68003520 con 0x7f6a780a8e00 2023-12-11T20:17:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a780a9c80 msgr2=0x7f6a78145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a780a9c80 0x7f6a78145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 >> v2:172.21.15.86:3300/0 conn(0x7f6a780a8a20 msgr2=0x7f6a781459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 --2- 172.21.15.86:0/1485518824 >> v2:172.21.15.86:3300/0 conn(0x7f6a780a8a20 0x7f6a781459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 monclient: found mon.noname-c-legacy 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.133:6789/0 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a780b2280 con 0x7f6a780a8e00 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1416536749 0 0) 0x7f6a700019f0 con 0x7f6a780a8e00 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a7ee89640 5 monclient: authenticate success, global_id 4432 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a7ee89640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: dump: 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] but i'm connected to v1:172.21.15.133:6789/0, reconnecting 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 >> v1:172.21.15.133:6789/0 conn(0x7f6a780a8e00 legacy=0x7f6a78141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:17:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a68003970 0x7f6a68003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient(hunting): picked mon.a con 0x7f6a68003970 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a680047c0 0x7f6a68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient(hunting): picked mon.b con 0x7f6a680047c0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a780a9c80 0x7f6a6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient(hunting): picked mon.c con 0x7f6a780a9c80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780aaf50 con 0x7f6a68003970 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a781454d0 con 0x7f6a680047c0 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68005670 con 0x7f6a780a9c80 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a777fe640 10 monclient(hunting): _finish_auth 0 2023-12-11T20:17:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.601+0000 7f6a7ee89640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.604+0000 7f6a77fff640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a680047c0 0x7f6a68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.604+0000 7f6a77fff640 10 monclient(hunting): get_auth_request con 0x7f6a680047c0 auth_method 0 2023-12-11T20:17:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.604+0000 7f6a77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.604+0000 7f6a77fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.604+0000 7f6a77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:17:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.605+0000 7f6a77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.605+0000 7f6a77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.605+0000 7f6a77fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:17:03.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 10 monclient(hunting): handle_auth_done global_id 4432 payload 293 2023-12-11T20:17:03.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 1 -- 172.21.15.86:0/1485518824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a780a9c80 msgr2=0x7f6a6800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a780a9c80 0x7f6a6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 1 -- 172.21.15.86:0/1485518824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a68003970 msgr2=0x7f6a68003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a68003970 0x7f6a68003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 1 monclient: found mon.b 2023-12-11T20:17:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 1 -- 172.21.15.86:0/1485518824 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a68003520 con 0x7f6a680047c0 2023-12-11T20:17:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.607+0000 7f6a77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.609+0000 7f6a77fff640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a680047c0 0x7f6a68004c40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009860 tx=0x7f6a680057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa70f334311dfdb4 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64032600 con 0x7f6a680047c0 2023-12-11T20:17:03.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:03.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 10 monclient: dump: 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6402e050 con 0x7f6a680047c0 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a7ee89640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a7ee89640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:03.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 1 -- 172.21.15.86:0/1485518824 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640329c0 con 0x7f6a680047c0 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a7ee89640 4 set_mon_vals no callback set 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.612+0000 7f6a777fe640 10 monclient: dump: 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.612 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.613 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.613 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7ee89640 10 monclient: shutdown 2023-12-11T20:17:03.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7ee89640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:03.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7ee89640 1 -- 172.21.15.86:0/1485518824 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a680047c0 msgr2=0x7f6a68004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7ee89640 1 --2- 172.21.15.86:0/1485518824 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a680047c0 0x7f6a68004c40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009860 tx=0x7f6a680057f0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7d426640 1 -- 172.21.15.86:0/1485518824 reap_dead start 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7ee89640 1 -- 172.21.15.86:0/1485518824 shutdown_connections 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7ee89640 1 -- 172.21.15.86:0/1485518824 >> 172.21.15.86:0/1485518824 conn(0x7f6a78061ac0 msgr2=0x7f6a780b7880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7ee89640 1 -- 172.21.15.86:0/1485518824 shutdown_connections 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.613+0000 7f6a7ee89640 1 -- 172.21.15.86:0/1485518824 wait complete. 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.614+0000 7f6a7ee89640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.614+0000 7f6a7ee89640 10 monclient: monmap: 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.614+0000 7f6a7ee89640 1 librados: starting msgr at 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.614+0000 7f6a7ee89640 1 librados: starting objecter 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.615+0000 7f6a7ee89640 1 Processor -- start 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.615+0000 7f6a7ee89640 1 -- start start 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.615+0000 7f6a7ee89640 1 librados: setting wanted keys 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.615+0000 7f6a7ee89640 1 librados: calling monclient init 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.615+0000 7f6a7ee89640 10 monclient: init 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a680047c0 0x7f6a781cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient(hunting): picked mon.noname-a con 0x7f6a680047c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a781389b0 0x7f6a78138df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient(hunting): picked mon.noname-c con 0x7f6a781389b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a781cc780 0x7f6a781ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient(hunting): picked mon.noname-b con 0x7f6a781cc780 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a781cc2f0 con 0x7f6a680047c0 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a781ce200 con 0x7f6a781cc780 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a781ce4e0 con 0x7f6a781389b0 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.616+0000 7f6a7ee89640 10 monclient(hunting): authenticate will time out at 1509.945607s 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a781cc780 0x7f6a781ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a781cc780 0x7f6a781ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52582/0 (socket says 172.21.15.86:52582) 2023-12-11T20:17:03.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 1 -- 172.21.15.86:0/2317679546 learned_addr learned my addr 172.21.15.86:0/2317679546 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 10 monclient(hunting): get_auth_request con 0x7f6a781cc780 auth_method 0 2023-12-11T20:17:03.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.617+0000 7f6a7d426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 10 monclient(hunting): handle_auth_done global_id 4435 payload 824 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 1 -- 172.21.15.86:0/2317679546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a781389b0 msgr2=0x7f6a78138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a781389b0 0x7f6a78138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 1 -- 172.21.15.86:0/2317679546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a680047c0 msgr2=0x7f6a781cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a680047c0 0x7f6a781cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 1 monclient: found mon.noname-b 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.620+0000 7f6a7d426640 1 -- 172.21.15.86:0/2317679546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a781cbe80 con 0x7f6a781cc780 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.622+0000 7f6a7d426640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a781cc780 0x7f6a781ccc00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6a78146470 tx=0x7f6a7000ccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.622+0000 7f6a757fa640 10 client.?.objecter ms_handle_connect 0x7f6a781cc780 2023-12-11T20:17:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.622+0000 7f6a757fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 1 -- 172.21.15.86:0/2317679546 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7000a610 con 0x7f6a781cc780 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 10 monclient: dump: 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.626 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 1 -- 172.21.15.86:0/2317679546 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7000a770 con 0x7f6a781cc780 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a7ee89640 5 monclient: authenticate success, global_id 4435 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a7ee89640 10 monclient: _renew_subs 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a7ee89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a7ee89640 1 -- 172.21.15.86:0/2317679546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a78139330 con 0x7f6a781cc780 2023-12-11T20:17:03.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a757fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a76ffd640 4 set_mon_vals no callback set 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.626+0000 7f6a7ee89640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.627+0000 7f6a757fa640 1 -- 172.21.15.86:0/2317679546 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7000ab20 con 0x7f6a781cc780 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.627+0000 7f6a7ee89640 10 monclient: _renew_subs 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.627+0000 7f6a7ee89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.627+0000 7f6a7ee89640 1 -- 172.21.15.86:0/2317679546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a78139870 con 0x7f6a781cc780 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.627+0000 7f6a7ee89640 1 librados: init done 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.627+0000 7f6a757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.627+0000 7f6a757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.627+0000 7f6a757fa640 10 monclient: dump: 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.629 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.628+0000 7f6a62ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.628+0000 7f6a62ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.628+0000 7f6a62ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.628+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2317679546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a50005080 con 0x7f6a781cc780 2023-12-11T20:17:03.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.645+0000 7f6a757fa640 1 -- 172.21.15.86:0/2317679546 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6a7000ac80 con 0x7f6a781cc780 2023-12-11T20:17:03.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.646+0000 7f6a757fa640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6a5c06c820 0x7f6a5c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.646+0000 7f6a7cc25640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6a5c06c820 0x7f6a5c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.646+0000 7f6a7cc25640 10 monclient: get_auth_request con 0x7f6a5c06c820 auth_method 0 2023-12-11T20:17:03.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.647+0000 7f6a757fa640 10 client.4435.objecter ms_handle_connect 0x7f6a5c06c820 2023-12-11T20:17:03.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.647+0000 7f6a7cc25640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6a5c06c820 0x7f6a5c06ece0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009690 tx=0x7f6a6c0046c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.662+0000 7f37077fe640 1 -- 172.21.15.86:0/2574444183 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3718007670 con 0x7f3720145ab0 2023-12-11T20:17:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.662+0000 7f37077fe640 10 client.4328.objecter ms_dispatch 0x7f3720000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.662+0000 7f37077fe640 3 client.4328.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.662+0000 7f37077fe640 3 client.4328.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.662+0000 7f37077fe640 20 client.4328.objecter dump_active .. 0 homeless 2023-12-11T20:17:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.662+0000 7f37077fe640 1 -- 172.21.15.86:0/2574444183 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3718054010 con 0x7f3720145ab0 2023-12-11T20:17:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.662+0000 7f37077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.662+0000 7f37077fe640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:03.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.704+0000 7fb20b89b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:03.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.704+0000 7fb20b89b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:03.702 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.704+0000 7fb20b89b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:03.703 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.704+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb20404e840 con 0x7fb2040a8e00 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.704+0000 7f6ae071d640 10 monclient: get_monmap_and_config 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.704+0000 7f6ae071d640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.704+0000 7f6ae071d640 10 monclient: monmap: 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:03.704976+0000 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:03.704976+0000 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:03.704 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 1 Processor -- start 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 1 -- start start 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient: init 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ad80a8a20 0x7f6ad80a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad80a8a20 addr v2:172.21.15.86:3300/0 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad813f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ad80b2280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f6ad813f860 0x7f6ad8145b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.705 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ad813f860 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:03.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad8145670 con 0x7f6ad80a8a20 2023-12-11T20:17:03.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad80aa990 con 0x7f6ad813f860 2023-12-11T20:17:03.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad80aac70 con 0x7f6ad80b2280 2023-12-11T20:17:03.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.706 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.705+0000 7f6ae071d640 10 monclient(hunting): authenticate will time out at 1510.035004s 2023-12-11T20:17:03.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.706+0000 7f6addcb8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad813f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.707 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.706+0000 7f6addcb8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad813f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40326/0 (socket says 172.21.15.86:40326) 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.706+0000 7f6addcb8640 1 -- 172.21.15.86:0/473177325 learned_addr learned my addr 172.21.15.86:0/473177325 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.706+0000 7f6addcb8640 10 monclient(hunting): get_auth_request con 0x7f6ad80b2280 auth_method 0 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.706+0000 7f6addcb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.706+0000 7f6addcb8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.706+0000 7f6addcb8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.707+0000 7f6addcb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.707+0000 7f6addcb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.707+0000 7f6addcb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 1 -- 172.21.15.86:0/473177325 >> v2:172.21.15.133:3300/0 conn(0x7f6ad813f860 msgr2=0x7f6ad8145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 1 --2- 172.21.15.86:0/473177325 >> v2:172.21.15.133:3300/0 conn(0x7f6ad813f860 0x7f6ad8145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 1 -- 172.21.15.86:0/473177325 >> v2:172.21.15.86:3300/0 conn(0x7f6ad80a8a20 msgr2=0x7f6ad80a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:03.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 1 --2- 172.21.15.86:0/473177325 >> v2:172.21.15.86:3300/0 conn(0x7f6ad80a8a20 0x7f6ad80a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 1 monclient: found mon.noname-b 2023-12-11T20:17:03.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.709+0000 7f6addcb8640 1 -- 172.21.15.86:0/473177325 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad80aaf50 con 0x7f6ad80b2280 2023-12-11T20:17:03.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.711+0000 7f6addcb8640 1 --2- 172.21.15.86:0/473177325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad813f320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80b3a10 tx=0x7f6ac802b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91fce8ed9e4f3e71 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 1 -- 172.21.15.86:0/473177325 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8031750 con 0x7f6ad80b2280 2023-12-11T20:17:03.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:03.713 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: dump: 2023-12-11T20:17:03.713 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.713 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.713 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.713 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 1 -- 172.21.15.86:0/473177325 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac80318b0 con 0x7f6ad80b2280 2023-12-11T20:17:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 5 monclient: authenticate success, global_id 4331 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 1 -- 172.21.15.86:0/473177325 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8031c40 con 0x7f6ad80b2280 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 4 set_mon_vals no callback set 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6add4b7640 10 monclient: dump: 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.716 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 10 monclient: shutdown 2023-12-11T20:17:03.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:03.720 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 1 -- 172.21.15.86:0/473177325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 msgr2=0x7f6ad813f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 1 --2- 172.21.15.86:0/473177325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad813f320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80b3a10 tx=0x7f6ac802b7f0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.715+0000 7f6ae071d640 1 -- 172.21.15.86:0/473177325 shutdown_connections 2023-12-11T20:17:03.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.716+0000 7f6ae071d640 1 --2- 172.21.15.86:0/473177325 >> v2:172.21.15.133:3300/0 conn(0x7f6ad813f860 0x7f6ad8145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.716+0000 7f6ae071d640 1 --2- 172.21.15.86:0/473177325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad813f320 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.716+0000 7f6ae071d640 1 --2- 172.21.15.86:0/473177325 >> v2:172.21.15.86:3300/0 conn(0x7f6ad80a8a20 0x7f6ad80a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.721 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.716+0000 7f6ae071d640 1 -- 172.21.15.86:0/473177325 >> 172.21.15.86:0/473177325 conn(0x7f6ad8061a10 msgr2=0x7f6ad80b7880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.716+0000 7f6ae071d640 1 -- 172.21.15.86:0/473177325 shutdown_connections 2023-12-11T20:17:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.716+0000 7f6ae071d640 1 -- 172.21.15.86:0/473177325 wait complete. 2023-12-11T20:17:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.716+0000 7f6ae071d640 10 monclient: build_initial_monmap 2023-12-11T20:17:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.716+0000 7f6ae071d640 10 monclient: monmap: 2023-12-11T20:17:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:03.723 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:03.723 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:03.723 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.723 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:03.723 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:03.723 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:03.724 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 librados: starting msgr at 2023-12-11T20:17:03.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 librados: starting objecter 2023-12-11T20:17:03.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 Processor -- start 2023-12-11T20:17:03.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 -- start start 2023-12-11T20:17:03.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 librados: setting wanted keys 2023-12-11T20:17:03.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 librados: calling monclient init 2023-12-11T20:17:03.724 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient: init 2023-12-11T20:17:03.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:03.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:17:03.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad80a8a20 0x7f6ad8139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad80a8a20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:03.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad8144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ad80b2280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:03.725 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6ad813f860 0x7f6ad8134990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ad813f860 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad813b480 con 0x7f6ad80a8a20 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad813a780 con 0x7f6ad813f860 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad813b660 con 0x7f6ad80b2280 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:03.726 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.717+0000 7f6ae071d640 10 monclient(hunting): authenticate will time out at 1510.047002s 2023-12-11T20:17:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.718+0000 7f6addcb8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad8144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.718+0000 7f6addcb8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad8144980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40328/0 (socket says 172.21.15.86:40328) 2023-12-11T20:17:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.718+0000 7f6addcb8640 1 -- 172.21.15.86:0/1670590070 learned_addr learned my addr 172.21.15.86:0/1670590070 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.718+0000 7f6addcb8640 10 monclient(hunting): get_auth_request con 0x7f6ad80b2280 auth_method 0 2023-12-11T20:17:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.718+0000 7f6addcb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.718+0000 7f6addcb8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.718+0000 7f6addcb8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.719+0000 7f6addcb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:03.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.719+0000 7f6addcb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:03.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.719+0000 7f6addcb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:03.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.721+0000 7f6addcb8640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2023-12-11T20:17:03.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.721+0000 7f6addcb8640 1 -- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6ad813f860 msgr2=0x7f6ad8134990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.721+0000 7f6addcb8640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6ad813f860 0x7f6ad8134990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.721+0000 7f6addcb8640 1 -- 172.21.15.86:0/1670590070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad80a8a20 msgr2=0x7f6ad8139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.728 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.721+0000 7f6addcb8640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad80a8a20 0x7f6ad8139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.721+0000 7f6addcb8640 10 monclient: _finish_hunting 0 2023-12-11T20:17:03.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.721+0000 7f6addcb8640 1 monclient: found mon.noname-c 2023-12-11T20:17:03.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.722+0000 7f6addcb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:03.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.722+0000 7f6addcb8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.729 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.722+0000 7f6addcb8640 1 -- 172.21.15.86:0/1670590070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac80094d0 con 0x7f6ad80b2280 2023-12-11T20:17:03.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.724+0000 7f6addcb8640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad8144980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8000c00 tx=0x7f6ac80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.724+0000 7f6aceffd640 10 client.?.objecter ms_handle_connect 0x7f6ad80b2280 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.724+0000 7f6aceffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 1 -- 172.21.15.86:0/1670590070 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8033c80 con 0x7f6ad80b2280 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: dump: 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.732 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.733 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.733 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: _finish_auth 0 2023-12-11T20:17:03.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: _check_auth_tickets 2023-12-11T20:17:03.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:03.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 1 -- 172.21.15.86:0/1670590070 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac8033de0 con 0x7f6ad80b2280 2023-12-11T20:17:03.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6adccb6640 4 set_mon_vals no callback set 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 5 monclient: authenticate success, global_id 4334 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 1 -- 172.21.15.86:0/1670590070 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac803c620 con 0x7f6ad80b2280 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6aceffd640 10 monclient: dump: 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:03.734 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 10 monclient: _renew_subs 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad813a260 con 0x7f6ad80b2280 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 10 monclient: _renew_subs 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad8134ed0 con 0x7f6ad80b2280 2023-12-11T20:17:03.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.728+0000 7f6ae071d640 1 librados: init done 2023-12-11T20:17:03.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.729+0000 7f6ae071d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.729+0000 7f6ae071d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.729+0000 7f6ae071d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.729+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad804e840 con 0x7f6ad80b2280 2023-12-11T20:17:03.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.748+0000 7f6aceffd640 1 -- 172.21.15.86:0/1670590070 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6ac803a040 con 0x7f6ad80b2280 2023-12-11T20:17:03.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.748+0000 7f6aceffd640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6abc06c870 0x7f6abc06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:03.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.748+0000 7f6ade4b9640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6abc06c870 0x7f6abc06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:03.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.748+0000 7f6ade4b9640 10 monclient: get_auth_request con 0x7f6abc06c870 auth_method 0 2023-12-11T20:17:03.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.749+0000 7f6aceffd640 10 client.4334.objecter ms_handle_connect 0x7f6abc06c870 2023-12-11T20:17:03.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.749+0000 7f6ade4b9640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6abc06c870 0x7f6abc06ed30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009710 tx=0x7f6ad4006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.772+0000 7fb678ff9640 1 -- 172.21.15.86:0/3887873985 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb67407bec0 con 0x7fb67c1461c0 2023-12-11T20:17:03.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.772+0000 7fb678ff9640 10 client.4365.objecter ms_dispatch 0x7fb67c000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:03.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.772+0000 7fb678ff9640 3 client.4365.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:03.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.772+0000 7fb678ff9640 3 client.4365.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:03.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.772+0000 7fb678ff9640 20 client.4365.objecter dump_active .. 0 homeless 2023-12-11T20:17:03.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.772+0000 7fb678ff9640 1 -- 172.21.15.86:0/3887873985 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6740aae60 con 0x7fb67c1461c0 2023-12-11T20:17:03.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.772+0000 7fb678ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.772+0000 7fb678ff9640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:03.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.848+0000 7f37250a2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:03.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.848+0000 7f37250a2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:03.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.848+0000 7f37250a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:03.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.848+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f37201d7000 con 0x7f3720145ab0 2023-12-11T20:17:03.868 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.870+0000 7f6a757fa640 1 -- 172.21.15.86:0/2317679546 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6a70058750 con 0x7f6a781cc780 2023-12-11T20:17:03.868 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.870+0000 7f6a757fa640 10 client.4435.objecter ms_dispatch 0x7f6a78000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:03.868 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.870+0000 7f6a757fa640 3 client.4435.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:03.868 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.870+0000 7f6a757fa640 3 client.4435.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:03.868 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.870+0000 7f6a757fa640 20 client.4435.objecter dump_active .. 0 homeless 2023-12-11T20:17:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.881+0000 7f6a757fa640 1 -- 172.21.15.86:0/2317679546 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a70054390 con 0x7f6a781cc780 2023-12-11T20:17:03.879 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.881+0000 7f6a757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:03.879 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.881+0000 7f6a757fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:03.892 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.894+0000 7fb1f9ffb640 1 -- 172.21.15.86:0/3264988789 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb1f404c090 con 0x7fb2040a8e00 2023-12-11T20:17:03.892 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.894+0000 7fb1f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-11T20:17:03.892 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.894+0000 7fb1f9ffb640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:60129542149 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 10 librados: watch_flush enter 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 10 librados: watch_flush exit 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 20 client.4429.objecter shutdown clearing up homeless session... 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 10 client.4429.objecter successfully canceled tick 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb1e806c870 msgr2=0x7fb1e806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb1e806c870 0x7fb1e806ed30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb200009bd0 tx=0x7fb200009240 comp rx=0 tx=0).stop 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 10 monclient: shutdown 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 msgr2=0x7fb20413aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413aeb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4000c00 tx=0x7fb1f40046c0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 shutdown_connections 2023-12-11T20:17:03.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb1e806c870 0x7fb1e806ed30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2041423b0 0x7fb2041d7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2040a8e00 0x7fb20413aeb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 --2- 172.21.15.86:0/3264988789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2040a8a20 0x7fb204146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 >> 172.21.15.86:0/3264988789 conn(0x7fb2040b7870 msgr2=0x7fb204142840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:03.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 shutdown_connections 2023-12-11T20:17:03.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 -- 172.21.15.86:0/3264988789 wait complete. 2023-12-11T20:17:03.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.896+0000 7fb20b89b640 1 librados: shutdown 2023-12-11T20:17:03.905 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.4 2023-12-11T20:17:03.905 DEBUG:teuthology.parallel:result is None 2023-12-11T20:17:03.986 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.988+0000 7fb682590640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:03.986 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.988+0000 7fb682590640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:03.986 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.988+0000 7fb682590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:03.987 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.988+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb67c13a8d0 con 0x7fb67c1461c0 2023-12-11T20:17:03.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.993+0000 7fb678ff9640 1 -- 172.21.15.86:0/3887873985 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb674043090 con 0x7fb67c1461c0 2023-12-11T20:17:03.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.993+0000 7fb678ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-11T20:17:03.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.993+0000 7fb678ff9640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 10 librados: watch_flush enter 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 10 librados: watch_flush exit 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 20 client.4365.objecter shutdown clearing up homeless session... 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 10 client.4365.objecter successfully canceled tick 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb66806c820 msgr2=0x7fb66806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb66806c820 0x7fb66806ece0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb66c001560 tx=0x7fb66c007990 comp rx=0 tx=0).stop 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 10 monclient: shutdown 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c1461c0 msgr2=0x7fb67c1dadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c1461c0 0x7fb67c1dadc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb674036040 tx=0x7fb674004a40 comp rx=0 tx=0).stop 2023-12-11T20:17:03.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 shutdown_connections 2023-12-11T20:17:03.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fb66806c820 0x7fb66806ece0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb67c13f0f0 0x7fb67c1da880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb67c1461c0 0x7fb67c1dadc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 --2- 172.21.15.86:0/3887873985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb67c0a8a20 0x7fb67c1441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:03.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 >> 172.21.15.86:0/3887873985 conn(0x7fb67c0b7890 msgr2=0x7fb67c0b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:03.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 shutdown_connections 2023-12-11T20:17:03.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 -- 172.21.15.86:0/3887873985 wait complete. 2023-12-11T20:17:03.995 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:03.995+0000 7fb682590640 1 librados: shutdown 2023-12-11T20:17:04.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.2 2023-12-11T20:17:04.005 DEBUG:teuthology.parallel:result is None 2023-12-11T20:17:04.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.008+0000 7f6aceffd640 1 -- 172.21.15.86:0/1670590070 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6ac807b080 con 0x7f6ad80b2280 2023-12-11T20:17:04.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.008+0000 7f6aceffd640 10 client.4334.objecter ms_dispatch 0x7f6ad8000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:04.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.008+0000 7f6aceffd640 3 client.4334.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:04.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.008+0000 7f6aceffd640 3 client.4334.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:04.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.008+0000 7f6aceffd640 20 client.4334.objecter dump_active .. 0 homeless 2023-12-11T20:17:04.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.008+0000 7f6aceffd640 1 -- 172.21.15.86:0/1670590070 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ac8076cc0 con 0x7f6ad80b2280 2023-12-11T20:17:04.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.008+0000 7f6aceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:04.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.008+0000 7f6aceffd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:04.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.015+0000 7f37077fe640 1 -- 172.21.15.86:0/2574444183 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f37201d7000 con 0x7f3720145ab0 2023-12-11T20:17:04.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.015+0000 7f37077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-11T20:17:04.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.015+0000 7f37077fe640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:68719476742 2023-12-11T20:17:04.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 10 librados: watch_flush enter 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 10 librados: watch_flush exit 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 20 client.4328.objecter shutdown clearing up homeless session... 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 10 client.4328.objecter successfully canceled tick 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f370c06c870 msgr2=0x7f370c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f370c06c870 0x7f370c06ed30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f371002bfc0 tx=0x7f3710030000 comp rx=0 tx=0).stop 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 10 monclient: shutdown 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720145ab0 msgr2=0x7f37201d72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720145ab0 0x7f37201d72e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f371800c3c0 tx=0x7f371800c880 comp rx=0 tx=0).stop 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 shutdown_connections 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f370c06c870 0x7f370c06ed30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720145ab0 0x7f37201d72e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f372013f800 0x7f3720143860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 --2- 172.21.15.86:0/2574444183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200a8a20 0x7f3720142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 >> 172.21.15.86:0/2574444183 conn(0x7f37200b7890 msgr2=0x7f37200b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:04.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 shutdown_connections 2023-12-11T20:17:04.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 -- 172.21.15.86:0/2574444183 wait complete. 2023-12-11T20:17:04.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.017+0000 7f37250a2640 1 librados: shutdown 2023-12-11T20:17:04.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.6 2023-12-11T20:17:04.027 DEBUG:teuthology.parallel:result is None 2023-12-11T20:17:04.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.130+0000 7f6a62ffd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:04.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.131+0000 7f6a62ffd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:04.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.131+0000 7f6a62ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.129 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.131+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2317679546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6a50005600 con 0x7f6a781cc780 2023-12-11T20:17:04.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.134+0000 7f6a757fa640 1 -- 172.21.15.86:0/2317679546 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a700579b0 con 0x7f6a781cc780 2023-12-11T20:17:04.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.134+0000 7f6a757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-11T20:17:04.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.134+0000 7f6a757fa640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:55834574853 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 10 librados: watch_flush enter 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 10 librados: watch_flush exit 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 20 client.4435.objecter shutdown clearing up homeless session... 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 10 client.4435.objecter successfully canceled tick 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6a5c06c820 msgr2=0x7f6a5c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6a5c06c820 0x7f6a5c06ece0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009690 tx=0x7f6a6c0046c0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 10 monclient: shutdown 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a781cc780 msgr2=0x7f6a781ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a781cc780 0x7f6a781ccc00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6a78146470 tx=0x7f6a7000ccd0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2317679546 shutdown_connections 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6a5c06c820 0x7f6a5c06ece0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a781389b0 0x7f6a78138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a781cc780 0x7f6a781ccc00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 --2- 172.21.15.86:0/2317679546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a680047c0 0x7f6a781cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2317679546 >> 172.21.15.86:0/2317679546 conn(0x7f6a78061ac0 msgr2=0x7f6a780b7070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:04.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2317679546 shutdown_connections 2023-12-11T20:17:04.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2317679546 wait complete. 2023-12-11T20:17:04.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.137+0000 7f6a62ffd640 1 librados: shutdown 2023-12-11T20:17:04.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2023-12-11T20:17:04.147 DEBUG:teuthology.parallel:result is None 2023-12-11T20:17:04.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.198+0000 7f6ae071d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:04.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.198+0000 7f6ae071d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:04.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.198+0000 7f6ae071d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:17:04.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.198+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6ad813a3f0 con 0x7f6ad80b2280 2023-12-11T20:17:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.247+0000 7f6aceffd640 1 -- 172.21.15.86:0/1670590070 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ac804d020 con 0x7f6ad80b2280 2023-12-11T20:17:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.247+0000 7f6aceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-11T20:17:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.247+0000 7f6aceffd640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:68719476743 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.249+0000 7f6ae071d640 10 librados: watch_flush enter 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 10 librados: watch_flush exit 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 20 client.4334.objecter shutdown clearing up homeless session... 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 10 client.4334.objecter successfully canceled tick 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6abc06c870 msgr2=0x7f6abc06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6abc06c870 0x7f6abc06ed30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009710 tx=0x7f6ad4006c90 comp rx=0 tx=0).stop 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 10 monclient: shutdown 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 msgr2=0x7f6ad8144980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad8144980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8000c00 tx=0x7f6ac80046c0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 shutdown_connections 2023-12-11T20:17:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f6abc06c870 0x7f6abc06ed30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad80b2280 0x7f6ad8144980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6ad813f860 0x7f6ad8134990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 --2- 172.21.15.86:0/1670590070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad80a8a20 0x7f6ad8139c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 >> 172.21.15.86:0/1670590070 conn(0x7f6ad8061a10 msgr2=0x7f6ad80b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 shutdown_connections 2023-12-11T20:17:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 -- 172.21.15.86:0/1670590070 wait complete. 2023-12-11T20:17:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.250+0000 7f6ae071d640 1 librados: shutdown 2023-12-11T20:17:04.260 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.0 2023-12-11T20:17:04.260 DEBUG:teuthology.parallel:result is None 2023-12-11T20:17:04.260 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-11T20:17:04.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.361+0000 7f120fafc640 10 monclient: get_monmap_and_config 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.361+0000 7f120fafc640 10 monclient: build_initial_monmap 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.361+0000 7f120fafc640 10 monclient: monmap: 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:04.361690+0000 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:04.361690+0000 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:04.360 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.361+0000 7f120fafc640 1 Processor -- start 2023-12-11T20:17:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.361+0000 7f120fafc640 1 -- start start 2023-12-11T20:17:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.361+0000 7f120fafc640 10 monclient: init 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f12080a8a20 0x7f12080a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 10 monclient(hunting): picked mon.noname-c con 0x7f12080a8a20 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1208145ab0 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f120813f800 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1208145620 con 0x7f120813f800 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12080aace0 con 0x7f1208145ab0 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12080aafc0 con 0x7f12080a8a20 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.362+0000 7f120fafc640 10 monclient(hunting): authenticate will time out at 1510.691767s 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.363+0000 7f120e099640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f120813f800 0x7f120814c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45376/0 (socket says 172.21.15.86:45376) 2023-12-11T20:17:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.363+0000 7f120e099640 1 -- 172.21.15.86:0/3045929680 learned_addr learned my addr 172.21.15.86:0/3045929680 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:17:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.363+0000 7f120d898640 1 --2- 172.21.15.86:0/3045929680 >> v2:172.21.15.133:3300/0 conn(0x7f12080a8a20 0x7f12080a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.363+0000 7f120d898640 10 monclient(hunting): get_auth_request con 0x7f12080a8a20 auth_method 0 2023-12-11T20:17:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.363+0000 7f120d898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.363+0000 7f120d898640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.363+0000 7f120d898640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.364+0000 7f120d898640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.364+0000 7f120d898640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.364+0000 7f120d898640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.365+0000 7f120c896640 1 -- 172.21.15.86:0/3045929680 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892610121 0 0) 0x7f1208145620 con 0x7f120813f800 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.365+0000 7f120c896640 10 monclient(hunting): _negotiate 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.365+0000 7f120c896640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.365+0000 7f120c896640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.365+0000 7f120c896640 10 monclient(hunting): my global_id is 4368 2023-12-11T20:17:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.365+0000 7f120c896640 1 -- 172.21.15.86:0/3045929680 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f4003520 con 0x7f120813f800 2023-12-11T20:17:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.368+0000 7f120d898640 10 monclient(hunting): handle_auth_done global_id 4438 payload 274 2023-12-11T20:17:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.368+0000 7f120d898640 1 -- 172.21.15.86:0/3045929680 >> v1:172.21.15.133:6789/0 conn(0x7f1208145ab0 legacy=0x7f120813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.368+0000 7f120d898640 1 -- 172.21.15.86:0/3045929680 >> v1:172.21.15.86:6789/0 conn(0x7f120813f800 legacy=0x7f120814c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.368+0000 7f120d898640 10 monclient: _finish_hunting 0 2023-12-11T20:17:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.368+0000 7f120d898640 1 monclient: found mon.noname-c 2023-12-11T20:17:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.368+0000 7f120d898640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.368+0000 7f120d898640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.368+0000 7f120d898640 1 -- 172.21.15.86:0/3045929680 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080b2210 con 0x7f12080a8a20 2023-12-11T20:17:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.371+0000 7f120d898640 1 --2- 172.21.15.86:0/3045929680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12080a8e00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1200009d90 tx=0x7f120002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d103cd085d0804d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120c896640 1 -- 172.21.15.86:0/3045929680 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120002cd00 con 0x7f12080a8a20 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120c896640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120c896640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120c896640 10 monclient: dump: 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:04.372 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120c896640 10 monclient: _finish_auth 0 2023-12-11T20:17:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120c896640 10 monclient: _check_auth_tickets 2023-12-11T20:17:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120fafc640 5 monclient: authenticate success, global_id 4438 2023-12-11T20:17:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.373+0000 7f120fafc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.374+0000 7f120c896640 1 -- 172.21.15.86:0/3045929680 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120002ce60 con 0x7f12080a8a20 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.374+0000 7f120c896640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.374+0000 7f120c896640 1 -- 172.21.15.86:0/3045929680 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200034640 con 0x7f12080a8a20 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.374+0000 7f120c896640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.374+0000 7f120c896640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.374+0000 7f120c896640 10 monclient: dump: 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:04.373 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.374+0000 7f120fafc640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.374+0000 7f120fafc640 4 set_mon_vals no callback set 2023-12-11T20:17:04.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 10 monclient: shutdown 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 1 -- 172.21.15.86:0/3045929680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 msgr2=0x7f12080a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 1 --2- 172.21.15.86:0/3045929680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12080a8e00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1200009d90 tx=0x7f120002bf50 comp rx=0 tx=0).stop 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 1 -- 172.21.15.86:0/3045929680 shutdown_connections 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 1 --2- 172.21.15.86:0/3045929680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12080a8e00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 1 -- 172.21.15.86:0/3045929680 >> 172.21.15.86:0/3045929680 conn(0x7f12080b7890 msgr2=0x7f12080b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 1 -- 172.21.15.86:0/3045929680 shutdown_connections 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.375+0000 7f120fafc640 1 -- 172.21.15.86:0/3045929680 wait complete. 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 10 monclient: build_initial_monmap 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 10 monclient: monmap: 2023-12-11T20:17:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 1 librados: starting msgr at 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 1 librados: starting objecter 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 1 Processor -- start 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 1 -- start start 2023-12-11T20:17:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 1 librados: setting wanted keys 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 1 librados: calling monclient init 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.376+0000 7f120fafc640 10 monclient: init 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12081daa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient(hunting): picked mon.noname-b con 0x7f12080a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f120813f800 0x7f1208140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient(hunting): picked mon.noname-a con 0x7f120813f800 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1208145ab0 0x7f1208142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient(hunting): picked mon.noname-c con 0x7f1208145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12081433a0 con 0x7f120813f800 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1208142fd0 con 0x7f12080a8a20 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1208143190 con 0x7f1208145ab0 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120fafc640 10 monclient(hunting): authenticate will time out at 1510.706394s 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120d097640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f120813f800 0x7f1208140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120d898640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12081daa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.377+0000 7f120e099640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1208145ab0 0x7f1208142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12081daa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52610/0 (socket says 172.21.15.86:52610) 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 1 -- 172.21.15.86:0/2566182569 learned_addr learned my addr 172.21.15.86:0/2566182569 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 10 monclient(hunting): get_auth_request con 0x7f12080a8a20 auth_method 0 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d097640 10 monclient(hunting): get_auth_request con 0x7f120813f800 auth_method 0 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d097640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d097640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d097640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120e099640 10 monclient(hunting): get_auth_request con 0x7f1208145ab0 auth_method 0 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120e099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120e099640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120e099640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d898640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d097640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d097640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120d097640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120e099640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120e099640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.378+0000 7f120e099640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 10 monclient(hunting): handle_auth_done global_id 4444 payload 824 2023-12-11T20:17:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 1 -- 172.21.15.86:0/2566182569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1208145ab0 msgr2=0x7f1208142a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1208145ab0 0x7f1208142a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 1 -- 172.21.15.86:0/2566182569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f120813f800 msgr2=0x7f1208140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f120813f800 0x7f1208140150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 10 monclient: _finish_hunting 0 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 1 monclient: found mon.noname-b 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.381+0000 7f120d898640 1 -- 172.21.15.86:0/2566182569 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1200009a70 con 0x7f12080a8a20 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.383+0000 7f120d898640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12081daa00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1200000f80 tx=0x7f1200004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.383+0000 7f11fe7fc640 10 client.?.objecter ms_handle_connect 0x7f12080a8a20 2023-12-11T20:17:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.383+0000 7f11fe7fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 1 -- 172.21.15.86:0/2566182569 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200004880 con 0x7f12080a8a20 2023-12-11T20:17:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: dump: 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: _finish_auth 0 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 1 -- 172.21.15.86:0/2566182569 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1200009450 con 0x7f12080a8a20 2023-12-11T20:17:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 1 -- 172.21.15.86:0/2566182569 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200040670 con 0x7f12080a8a20 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 5 monclient: authenticate success, global_id 4444 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 10 monclient: _renew_subs 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12081db4b0 con 0x7f12080a8a20 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fffff640 4 set_mon_vals no callback set 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f11fe7fc640 10 monclient: dump: 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 10 monclient: _renew_subs 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1208143aa0 con 0x7f12080a8a20 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.387+0000 7f120fafc640 1 librados: init done 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.388+0000 7f120fafc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.388+0000 7f120fafc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.388+0000 7f120fafc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.388+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120804e840 con 0x7f12080a8a20 2023-12-11T20:17:04.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.407+0000 7f11fe7fc640 1 -- 172.21.15.86:0/2566182569 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f12000407d0 con 0x7f12080a8a20 2023-12-11T20:17:04.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.407+0000 7f11fe7fc640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f11ec06c8a0 0x7f11ec06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.407+0000 7f120d097640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f11ec06c8a0 0x7f11ec06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.407+0000 7f120d097640 10 monclient: get_auth_request con 0x7f11ec06c8a0 auth_method 0 2023-12-11T20:17:04.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.408+0000 7f120d097640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f11ec06c8a0 0x7f11ec06ed60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1208141b30 tx=0x7f11f8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:04.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.408+0000 7f11fe7fc640 10 client.4444.objecter ms_handle_connect 0x7f11ec06c8a0 2023-12-11T20:17:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.635+0000 7f11fe7fc640 1 -- 172.21.15.86:0/2566182569 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f120007b450 con 0x7f12080a8a20 2023-12-11T20:17:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.635+0000 7f11fe7fc640 10 client.4444.objecter ms_dispatch 0x7f1208000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.635+0000 7f11fe7fc640 3 client.4444.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.635+0000 7f11fe7fc640 3 client.4444.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.635+0000 7f11fe7fc640 20 client.4444.objecter dump_active .. 0 homeless 2023-12-11T20:17:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.646+0000 7f11fe7fc640 1 -- 172.21.15.86:0/2566182569 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1200077790 con 0x7f12080a8a20 2023-12-11T20:17:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.646+0000 7f11fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.646+0000 7f11fe7fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:04.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.815+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f120813a8f0 con 0x7f11ec06c8a0 2023-12-11T20:17:04.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.817+0000 7f11fe7fc640 1 -- 172.21.15.86:0/2566182569 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48073 (secure 0 0 0) 0x7f120813a8f0 con 0x7f11ec06c8a0 2023-12-11T20:17:04.816 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:17:04.816 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 10 librados: watch_flush enter 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 10 librados: watch_flush exit 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 20 client.4444.objecter shutdown clearing up homeless session... 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 10 client.4444.objecter successfully canceled tick 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f11ec06c8a0 msgr2=0x7f11ec06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f11ec06c8a0 0x7f11ec06ed60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1208141b30 tx=0x7f11f8009040 comp rx=0 tx=0).stop 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 10 monclient: shutdown 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 msgr2=0x7f12081daa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12081daa00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1200000f80 tx=0x7f1200004a40 comp rx=0 tx=0).stop 2023-12-11T20:17:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 shutdown_connections 2023-12-11T20:17:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f11ec06c8a0 0x7f11ec06ed60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1208145ab0 0x7f1208142a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f120813f800 0x7f1208140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 --2- 172.21.15.86:0/2566182569 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12080a8a20 0x7f12081daa00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 >> 172.21.15.86:0/2566182569 conn(0x7f12080b7890 msgr2=0x7f12080b7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 shutdown_connections 2023-12-11T20:17:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 -- 172.21.15.86:0/2566182569 wait complete. 2023-12-11T20:17:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.819+0000 7f120fafc640 1 librados: shutdown 2023-12-11T20:17:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-12-11T20:17:04.537659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2480,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2539520,"data_stored":1390240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001227"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.405804+0000","last_change":"2023-12-11T20:16:58.076986+0000","last_active":"2023-12-11T20:16:59.405804+0000","last_peered":"2023-12-11T20:16:59.405804+0000","last_clean":"2023-12-11T20:16:59.405804+0000","last_became_active":"2023-12-11T20:16:58.071647+0000","last_became_peered":"2023-12-11T20:16:58.071647+0000","last_unstale":"2023-12-11T20:16:59.405804+0000","last_undegraded":"2023-12-11T20:16:59.405804+0000","last_fullsized":"2023-12-11T20:16:59.405804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:29:07.975896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:58.816437+0000","last_change":"2023-12-11T20:16:57.283249+0000","last_active":"2023-12-11T20:16:58.816437+0000","last_peered":"2023-12-11T20:16:58.816437+0000","last_clean":"2023-12-11T20:16:58.816437+0000","last_became_active":"2023-12-11T20:16:57.255904+0000","last_became_peered":"2023-12-11T20:16:57.255904+0000","last_unstale":"2023-12-11T20:16:58.816437+0000","last_undegraded":"2023-12-11T20:16:58.816437+0000","last_fullsized":"2023-12-11T20:16:58.816437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:17:34.118238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.399853+0000","last_change":"2023-12-11T20:16:58.680128+0000","last_active":"2023-12-11T20:16:59.399853+0000","last_peered":"2023-12-11T20:16:59.399853+0000","last_clean":"2023-12-11T20:16:59.399853+0000","last_became_active":"2023-12-11T20:16:58.674889+0000","last_became_peered":"2023-12-11T20:16:58.674889+0000","last_unstale":"2023-12-11T20:16:59.399853+0000","last_undegraded":"2023-12-11T20:16:59.399853+0000","last_fullsized":"2023-12-11T20:16:59.399853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:54:50.773022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.400896+0000","last_change":"2023-12-11T20:16:58.412827+0000","last_active":"2023-12-11T20:16:59.400896+0000","last_peered":"2023-12-11T20:16:59.400896+0000","last_clean":"2023-12-11T20:16:59.400896+0000","last_became_active":"2023-12-11T20:16:58.383357+0000","last_became_peered":"2023-12-11T20:16:58.383357+0000","last_unstale":"2023-12-11T20:16:59.400896+0000","last_undegraded":"2023-12-11T20:16:59.400896+0000","last_fullsized":"2023-12-11T20:16:59.400896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T20:59:10.927708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.393460+0000","last_change":"2023-12-11T20:16:58.356627+0000","last_active":"2023-12-11T20:16:59.393460+0000","last_peered":"2023-12-11T20:16:59.393460+0000","last_clean":"2023-12-11T20:16:59.393460+0000","last_became_active":"2023-12-11T20:16:58.349599+0000","last_became_peered":"2023-12-11T20:16:58.349599+0000","last_unstale":"2023-12-11T20:16:59.393460+0000","last_undegraded":"2023-12-11T20:16:59.393460+0000","last_fullsized":"2023-12-11T20:16:59.393460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:45:43.507550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.388922+0000","last_change":"2023-12-11T20:16:58.236956+0000","last_active":"2023-12-11T20:16:59.388922+0000","last_peered":"2023-12-11T20:16:59.388922+0000","last_clean":"2023-12-11T20:16:59.388922+0000","last_became_active":"2023-12-11T20:16:58.208987+0000","last_became_peered":"2023-12-11T20:16:58.208987+0000","last_unstale":"2023-12-11T20:16:59.388922+0000","last_undegraded":"2023-12-11T20:16:59.388922+0000","last_fullsized":"2023-12-11T20:16:59.388922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:57:37.650191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.380894+0000","last_change":"2023-12-11T20:16:58.036508+0000","last_active":"2023-12-11T20:16:59.380894+0000","last_peered":"2023-12-11T20:16:59.380894+0000","last_clean":"2023-12-11T20:16:59.380894+0000","last_became_active":"2023-12-11T20:16:58.008642+0000","last_became_peered":"2023-12-11T20:16:58.008642+0000","last_unstale":"2023-12-11T20:16:59.380894+0000","last_undegraded":"2023-12-11T20:16:59.380894+0000","last_fullsized":"2023-12-11T20:16:59.380894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:46:42.050513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.396624+0000","last_change":"2023-12-11T20:16:58.643847+0000","last_active":"2023-12-11T20:16:59.396624+0000","last_peered":"2023-12-11T20:16:59.396624+0000","last_clean":"2023-12-11T20:16:59.396624+0000","last_became_active":"2023-12-11T20:16:58.638544+0000","last_became_peered":"2023-12-11T20:16:58.638544+0000","last_unstale":"2023-12-11T20:16:59.396624+0000","last_undegraded":"2023-12-11T20:16:59.396624+0000","last_fullsized":"2023-12-11T20:16:59.396624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:17:51.003853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-11T20:16:58.637710+0000","last_change":"2023-12-11T20:16:49.468937+0000","last_active":"2023-12-11T20:16:58.637710+0000","last_peered":"2023-12-11T20:16:58.637710+0000","last_clean":"2023-12-11T20:16:58.637710+0000","last_became_active":"2023-12-11T20:16:49.435576+0000","last_became_peered":"2023-12-11T20:16:49.435576+0000","last_unstale":"2023-12-11T20:16:58.637710+0000","last_undegraded":"2023-12-11T20:16:58.637710+0000","last_fullsized":"2023-12-11T20:16:58.637710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_clean_scrub_stamp":"2023-12-11T20:16:45.844700+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:49:09.602489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719476743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.964}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2729999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.427}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9220000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.59}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5869999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.801}]}]},{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.484000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.949}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.874000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.027999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.207000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.562999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.398}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2830000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.195}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7119999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.6}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.606}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.212999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.603999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":618496,"data_stored":472160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6150000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.484}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8360000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.391}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.946}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.347000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.276999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.695}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4130000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.998000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.311999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.728000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.966000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.649000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5879999999999992}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.201000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2649999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9890000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.315999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.092000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.83}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.401999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0010000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2789999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9730000000000008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.19}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.542}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.521999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":618496,"data_stored":472160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8810000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.284000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.042}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.723000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.685}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.193000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.053999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-11T20:17:04.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-11T20:17:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.931+0000 7f7054c56640 10 monclient: get_monmap_and_config 2023-12-11T20:17:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.931+0000 7f7054c56640 10 monclient: build_initial_monmap 2023-12-11T20:17:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.931+0000 7f7054c56640 10 monclient: monmap: 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:04.931600+0000 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:04.931600+0000 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:04.930 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.931 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.931+0000 7f7054c56640 1 Processor -- start 2023-12-11T20:17:04.931 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.931+0000 7f7054c56640 1 -- start start 2023-12-11T20:17:04.931 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.931+0000 7f7054c56640 10 monclient: init 2023-12-11T20:17:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-12-11T20:17:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70500a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70501461c0 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8e00 0x7f705013f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 10 monclient(hunting): picked mon.noname-b con 0x7f70500a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7050145620 con 0x7f70501461c0 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70500aace0 con 0x7f70500a8a20 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70500aafc0 con 0x7f70500a8e00 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.932+0000 7f7054c56640 10 monclient(hunting): authenticate will time out at 1511.261633s 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.933+0000 7f704ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8e00 0x7f705013f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.933+0000 7f704dd9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f70501461c0 0x7f705013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45378/0 (socket says 172.21.15.86:45378) 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.933+0000 7f704dd9b640 1 -- 172.21.15.86:0/954426482 learned_addr learned my addr 172.21.15.86:0/954426482 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.933+0000 7f704ed9d640 10 monclient(hunting): get_auth_request con 0x7f70500a8e00 auth_method 0 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.933+0000 7f704ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.933+0000 7f704ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.933+0000 7f704ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.934+0000 7f704ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.934+0000 7f704ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.934+0000 7f704ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 1 -- 172.21.15.86:0/954426482 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544962068 0 0) 0x7f70500aace0 con 0x7f70500a8a20 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 10 monclient(hunting): _negotiate 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 10 monclient(hunting): my global_id is 4447 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 1 -- 172.21.15.86:0/954426482 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7038003520 con 0x7f70500a8a20 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 1 -- 172.21.15.86:0/954426482 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899497010 0 0) 0x7f7050145620 con 0x7f70501461c0 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 10 monclient(hunting): _negotiate 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 10 monclient(hunting): my global_id is 4374 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704d59a640 1 -- 172.21.15.86:0/954426482 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70500aace0 con 0x7f70501461c0 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.935+0000 7f704ed9d640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.936+0000 7f704ed9d640 1 -- 172.21.15.86:0/954426482 >> v1:172.21.15.133:6789/0 conn(0x7f70500a8a20 legacy=0x7f7050145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.936+0000 7f704ed9d640 1 -- 172.21.15.86:0/954426482 >> v1:172.21.15.86:6789/0 conn(0x7f70501461c0 legacy=0x7f705013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.936+0000 7f704ed9d640 10 monclient: _finish_hunting 0 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.936+0000 7f704ed9d640 1 monclient: found mon.noname-b 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.936+0000 7f704ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.936+0000 7f704ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.936+0000 7f704ed9d640 1 -- 172.21.15.86:0/954426482 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70500b2210 con 0x7f70500a8e00 2023-12-11T20:17:04.937 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.937+0000 7f704ed9d640 1 --2- 172.21.15.86:0/954426482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8e00 0x7f705013f800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f70480097d0 tx=0x7f704802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b578679f91d759f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 1 -- 172.21.15.86:0/954426482 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048031890 con 0x7f70500a8e00 2023-12-11T20:17:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: dump: 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: _finish_auth 0 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: _check_auth_tickets 2023-12-11T20:17:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 1 -- 172.21.15.86:0/954426482 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70480319f0 con 0x7f70500a8e00 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f7054c56640 5 monclient: authenticate success, global_id 4340 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f7054c56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f7054c56640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f7054c56640 4 set_mon_vals no callback set 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 1 -- 172.21.15.86:0/954426482 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048031d80 con 0x7f70500a8e00 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.941+0000 7f704d59a640 10 monclient: dump: 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:04.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 10 monclient: shutdown 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 1 -- 172.21.15.86:0/954426482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8e00 msgr2=0x7f705013f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 1 --2- 172.21.15.86:0/954426482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8e00 0x7f705013f800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f70480097d0 tx=0x7f704802b930 comp rx=0 tx=0).stop 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 1 -- 172.21.15.86:0/954426482 shutdown_connections 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 1 --2- 172.21.15.86:0/954426482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8e00 0x7f705013f800 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 1 -- 172.21.15.86:0/954426482 >> 172.21.15.86:0/954426482 conn(0x7f70500b7890 msgr2=0x7f70500b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 1 -- 172.21.15.86:0/954426482 shutdown_connections 2023-12-11T20:17:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.942+0000 7f7054c56640 1 -- 172.21.15.86:0/954426482 wait complete. 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 10 monclient: build_initial_monmap 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 10 monclient: monmap: 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:04.946 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 1 librados: starting msgr at 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 1 librados: starting objecter 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 1 Processor -- start 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 1 -- start start 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 1 librados: setting wanted keys 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 1 librados: calling monclient init 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.943+0000 7f7054c56640 10 monclient: init 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8a20 0x7f70500b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient(hunting): picked mon.noname-c con 0x7f70500a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70500a8e00 0x7f7050140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient(hunting): picked mon.noname-a con 0x7f70500a8e00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70501461c0 0x7f7050138be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient(hunting): picked mon.noname-b con 0x7f70501461c0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7050138450 con 0x7f70500a8e00 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7050140ad0 con 0x7f70501461c0 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70501386c0 con 0x7f70500a8a20 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f7054c56640 10 monclient(hunting): authenticate will time out at 1511.273606s 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.944+0000 7f704e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8a20 0x7f70500b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704dd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70500a8e00 0x7f7050140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70501461c0 0x7f7050138be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70501461c0 0x7f7050138be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52612/0 (socket says 172.21.15.86:52612) 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 1 -- 172.21.15.86:0/4100512982 learned_addr learned my addr 172.21.15.86:0/4100512982 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 10 monclient(hunting): get_auth_request con 0x7f70501461c0 auth_method 0 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704e59c640 10 monclient(hunting): get_auth_request con 0x7f70500a8a20 auth_method 0 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704e59c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704e59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704dd9b640 10 monclient(hunting): get_auth_request con 0x7f70500a8e00 auth_method 0 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.945+0000 7f704e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.946+0000 7f704dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.946+0000 7f704dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.946+0000 7f704dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 10 monclient(hunting): handle_auth_done global_id 4450 payload 824 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 1 -- 172.21.15.86:0/4100512982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8a20 msgr2=0x7f70500b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8a20 0x7f70500b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 1 -- 172.21.15.86:0/4100512982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70500a8e00 msgr2=0x7f7050140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70500a8e00 0x7f7050140ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 10 monclient: _finish_hunting 0 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 1 monclient: found mon.noname-b 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.948+0000 7f704ed9d640 1 -- 172.21.15.86:0/4100512982 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7048009400 con 0x7f70501461c0 2023-12-11T20:17:04.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.950+0000 7f704ed9d640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70501461c0 0x7f7050138be0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f70480097d0 tx=0x7f70480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:04.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.950+0000 7f703effd640 10 client.?.objecter ms_handle_connect 0x7f70501461c0 2023-12-11T20:17:04.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.950+0000 7f703effd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:04.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 1 -- 172.21.15.86:0/4100512982 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704803bbb0 con 0x7f70501461c0 2023-12-11T20:17:04.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:04.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:04.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: dump: 2023-12-11T20:17:04.953 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:04.953 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: _finish_auth 0 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: _check_auth_tickets 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 1 -- 172.21.15.86:0/4100512982 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704803bd10 con 0x7f70501461c0 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 1 -- 172.21.15.86:0/4100512982 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048045630 con 0x7f70501461c0 2023-12-11T20:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f703effd640 10 monclient: dump: 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f704cd99640 4 set_mon_vals no callback set 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 5 monclient: authenticate success, global_id 4450 2023-12-11T20:17:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 10 monclient: _renew_subs 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7050141430 con 0x7f70501461c0 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 10 monclient: _renew_subs 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7050139120 con 0x7f70501461c0 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.954+0000 7f7054c56640 1 librados: init done 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.955+0000 7f7054c56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.955+0000 7f7054c56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.955+0000 7f7054c56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:04.955+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7050140920 con 0x7f70501461c0 2023-12-11T20:17:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.193+0000 7f703effd640 1 -- 172.21.15.86:0/4100512982 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f704804e450 con 0x7f70501461c0 2023-12-11T20:17:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.194+0000 7f703effd640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f703006c820 0x7f703006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.194+0000 7f704e59c640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f703006c820 0x7f703006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.194+0000 7f704e59c640 10 monclient: get_auth_request con 0x7f703006c820 auth_method 0 2023-12-11T20:17:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.195+0000 7f703effd640 10 client.4450.objecter ms_handle_connect 0x7f703006c820 2023-12-11T20:17:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.195+0000 7f704e59c640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f703006c820 0x7f703006ece0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7040007fd0 tx=0x7f7040009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.196+0000 7f703effd640 1 -- 172.21.15.86:0/4100512982 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f704807ad20 con 0x7f70501461c0 2023-12-11T20:17:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.196+0000 7f703effd640 10 client.4450.objecter ms_dispatch 0x7f7050000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.196+0000 7f703effd640 3 client.4450.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.196+0000 7f703effd640 3 client.4450.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.196+0000 7f703effd640 20 client.4450.objecter dump_active .. 0 homeless 2023-12-11T20:17:05.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.206+0000 7f703effd640 1 -- 172.21.15.86:0/4100512982 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7048077060 con 0x7f70501461c0 2023-12-11T20:17:05.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.206+0000 7f703effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:05.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.206+0000 7f703effd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.393+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70501415f0 con 0x7f703006c820 2023-12-11T20:17:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.396+0000 7f703effd640 1 -- 172.21.15.86:0/4100512982 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48073 (secure 0 0 0) 0x7f704000fe00 con 0x7f703006c820 2023-12-11T20:17:05.394 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:17:05.395 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-11T20:17:05.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 10 librados: watch_flush enter 2023-12-11T20:17:05.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 10 librados: watch_flush exit 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 20 client.4450.objecter shutdown clearing up homeless session... 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 10 client.4450.objecter successfully canceled tick 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f703006c820 msgr2=0x7f703006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f703006c820 0x7f703006ece0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7040007fd0 tx=0x7f7040009040 comp rx=0 tx=0).stop 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 10 monclient: shutdown 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70501461c0 msgr2=0x7f7050138be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70501461c0 0x7f7050138be0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f70480097d0 tx=0x7f70480046c0 comp rx=0 tx=0).stop 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 shutdown_connections 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f703006c820 0x7f703006ece0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70501461c0 0x7f7050138be0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70500a8e00 0x7f7050140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 --2- 172.21.15.86:0/4100512982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70500a8a20 0x7f70500b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 >> 172.21.15.86:0/4100512982 conn(0x7f70500b7890 msgr2=0x7f70500b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:05.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 shutdown_connections 2023-12-11T20:17:05.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 -- 172.21.15.86:0/4100512982 wait complete. 2023-12-11T20:17:05.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.398+0000 7f7054c56640 1 librados: shutdown 2023-12-11T20:17:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-12-11T20:17:04.537659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2480,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2539520,"data_stored":1390240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001227"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.405804+0000","last_change":"2023-12-11T20:16:58.076986+0000","last_active":"2023-12-11T20:16:59.405804+0000","last_peered":"2023-12-11T20:16:59.405804+0000","last_clean":"2023-12-11T20:16:59.405804+0000","last_became_active":"2023-12-11T20:16:58.071647+0000","last_became_peered":"2023-12-11T20:16:58.071647+0000","last_unstale":"2023-12-11T20:16:59.405804+0000","last_undegraded":"2023-12-11T20:16:59.405804+0000","last_fullsized":"2023-12-11T20:16:59.405804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:29:07.975896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:58.816437+0000","last_change":"2023-12-11T20:16:57.283249+0000","last_active":"2023-12-11T20:16:58.816437+0000","last_peered":"2023-12-11T20:16:58.816437+0000","last_clean":"2023-12-11T20:16:58.816437+0000","last_became_active":"2023-12-11T20:16:57.255904+0000","last_became_peered":"2023-12-11T20:16:57.255904+0000","last_unstale":"2023-12-11T20:16:58.816437+0000","last_undegraded":"2023-12-11T20:16:58.816437+0000","last_fullsized":"2023-12-11T20:16:58.816437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:17:34.118238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.399853+0000","last_change":"2023-12-11T20:16:58.680128+0000","last_active":"2023-12-11T20:16:59.399853+0000","last_peered":"2023-12-11T20:16:59.399853+0000","last_clean":"2023-12-11T20:16:59.399853+0000","last_became_active":"2023-12-11T20:16:58.674889+0000","last_became_peered":"2023-12-11T20:16:58.674889+0000","last_unstale":"2023-12-11T20:16:59.399853+0000","last_undegraded":"2023-12-11T20:16:59.399853+0000","last_fullsized":"2023-12-11T20:16:59.399853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:54:50.773022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.400896+0000","last_change":"2023-12-11T20:16:58.412827+0000","last_active":"2023-12-11T20:16:59.400896+0000","last_peered":"2023-12-11T20:16:59.400896+0000","last_clean":"2023-12-11T20:16:59.400896+0000","last_became_active":"2023-12-11T20:16:58.383357+0000","last_became_peered":"2023-12-11T20:16:58.383357+0000","last_unstale":"2023-12-11T20:16:59.400896+0000","last_undegraded":"2023-12-11T20:16:59.400896+0000","last_fullsized":"2023-12-11T20:16:59.400896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T20:59:10.927708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.393460+0000","last_change":"2023-12-11T20:16:58.356627+0000","last_active":"2023-12-11T20:16:59.393460+0000","last_peered":"2023-12-11T20:16:59.393460+0000","last_clean":"2023-12-11T20:16:59.393460+0000","last_became_active":"2023-12-11T20:16:58.349599+0000","last_became_peered":"2023-12-11T20:16:58.349599+0000","last_unstale":"2023-12-11T20:16:59.393460+0000","last_undegraded":"2023-12-11T20:16:59.393460+0000","last_fullsized":"2023-12-11T20:16:59.393460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:45:43.507550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.388922+0000","last_change":"2023-12-11T20:16:58.236956+0000","last_active":"2023-12-11T20:16:59.388922+0000","last_peered":"2023-12-11T20:16:59.388922+0000","last_clean":"2023-12-11T20:16:59.388922+0000","last_became_active":"2023-12-11T20:16:58.208987+0000","last_became_peered":"2023-12-11T20:16:58.208987+0000","last_unstale":"2023-12-11T20:16:59.388922+0000","last_undegraded":"2023-12-11T20:16:59.388922+0000","last_fullsized":"2023-12-11T20:16:59.388922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:57:37.650191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.380894+0000","last_change":"2023-12-11T20:16:58.036508+0000","last_active":"2023-12-11T20:16:59.380894+0000","last_peered":"2023-12-11T20:16:59.380894+0000","last_clean":"2023-12-11T20:16:59.380894+0000","last_became_active":"2023-12-11T20:16:58.008642+0000","last_became_peered":"2023-12-11T20:16:58.008642+0000","last_unstale":"2023-12-11T20:16:59.380894+0000","last_undegraded":"2023-12-11T20:16:59.380894+0000","last_fullsized":"2023-12-11T20:16:59.380894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:46:42.050513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-11T20:16:59.396624+0000","last_change":"2023-12-11T20:16:58.643847+0000","last_active":"2023-12-11T20:16:59.396624+0000","last_peered":"2023-12-11T20:16:59.396624+0000","last_clean":"2023-12-11T20:16:59.396624+0000","last_became_active":"2023-12-11T20:16:58.638544+0000","last_became_peered":"2023-12-11T20:16:58.638544+0000","last_unstale":"2023-12-11T20:16:59.396624+0000","last_undegraded":"2023-12-11T20:16:59.396624+0000","last_fullsized":"2023-12-11T20:16:59.396624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:17:51.003853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-11T20:16:58.637710+0000","last_change":"2023-12-11T20:16:49.468937+0000","last_active":"2023-12-11T20:16:58.637710+0000","last_peered":"2023-12-11T20:16:58.637710+0000","last_clean":"2023-12-11T20:16:58.637710+0000","last_became_active":"2023-12-11T20:16:49.435576+0000","last_became_peered":"2023-12-11T20:16:49.435576+0000","last_unstale":"2023-12-11T20:16:58.637710+0000","last_undegraded":"2023-12-11T20:16:58.637710+0000","last_fullsized":"2023-12-11T20:16:58.637710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_clean_scrub_stamp":"2023-12-11T20:16:45.844700+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:49:09.602489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719476743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.964}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2729999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.427}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9220000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.59}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5869999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.801}]}]},{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.484000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.949}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.874000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.027999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.207000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.562999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.398}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2830000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.195}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7119999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.6}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.606}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.212999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.603999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":618496,"data_stored":472160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6150000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.484}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8360000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.391}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.946}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.347000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.276999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.695}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4130000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.998000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.311999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.728000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.966000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.649000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5879999999999992}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.201000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2649999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9890000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.315999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.092000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.83}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.401999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0010000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2789999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9730000000000008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.19}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.542}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.521999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":618496,"data_stored":472160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8810000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.284000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.042}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.723000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.685}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.193000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.053999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-11T20:17:05.409 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-11T20:17:05.409 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-11T20:17:05.409 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-11T20:17:05.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-11T20:17:05.507 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.509+0000 7fde9f04b640 10 monclient: get_monmap_and_config 2023-12-11T20:17:05.507 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.509+0000 7fde9f04b640 10 monclient: build_initial_monmap 2023-12-11T20:17:05.507 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.509+0000 7fde9f04b640 10 monclient: monmap: 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:17:05.510087+0000 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:17:05.510087+0000 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:17:05.508 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.510+0000 7fde9f04b640 1 Processor -- start 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.510+0000 7fde9f04b640 1 -- start start 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.510+0000 7fde9f04b640 10 monclient: init 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.510+0000 7fde9f04b640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.510+0000 7fde9f04b640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.510+0000 7fde9f04b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde980719d0 addr v1:172.21.15.86:6789/0 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fde9813e2f0 0x7fde9813b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 10 monclient(hunting): picked mon.noname-c con 0x7fde9813e2f0 addr v2:172.21.15.133:3300/0 2023-12-11T20:17:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde98146bf0 addr v1:172.21.15.133:6789/0 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9813f1c0 con 0x7fde980719d0 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde98070ca0 con 0x7fde98146bf0 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9806f5b0 con 0x7fde9813e2f0 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9f04b640 10 monclient(hunting): authenticate will time out at 1511.840232s 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9cde7640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fde980719d0 0x7fde9806eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45394/0 (socket says 172.21.15.86:45394) 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde9cde7640 1 -- 172.21.15.86:0/2017795029 learned_addr learned my addr 172.21.15.86:0/2017795029 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.511+0000 7fde97fff640 1 --2- 172.21.15.86:0/2017795029 >> v2:172.21.15.133:3300/0 conn(0x7fde9813e2f0 0x7fde9813b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.512+0000 7fde97fff640 10 monclient(hunting): get_auth_request con 0x7fde9813e2f0 auth_method 0 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.512+0000 7fde97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.512+0000 7fde97fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.512+0000 7fde97fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.513+0000 7fde97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.513+0000 7fde97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.513+0000 7fde97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.514+0000 7fde977fe640 1 -- 172.21.15.86:0/2017795029 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159869 0 0) 0x7fde9813f1c0 con 0x7fde980719d0 2023-12-11T20:17:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.514+0000 7fde977fe640 10 monclient(hunting): _negotiate 2023-12-11T20:17:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.514+0000 7fde977fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.514+0000 7fde977fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.514+0000 7fde977fe640 10 monclient(hunting): my global_id is 4380 2023-12-11T20:17:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.514+0000 7fde977fe640 1 -- 172.21.15.86:0/2017795029 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde88003520 con 0x7fde980719d0 2023-12-11T20:17:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.515+0000 7fde97fff640 10 monclient(hunting): handle_auth_done global_id 4453 payload 274 2023-12-11T20:17:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.515+0000 7fde97fff640 1 -- 172.21.15.86:0/2017795029 >> v1:172.21.15.133:6789/0 conn(0x7fde98146bf0 legacy=0x7fde98148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.515+0000 7fde97fff640 1 -- 172.21.15.86:0/2017795029 >> v1:172.21.15.86:6789/0 conn(0x7fde980719d0 legacy=0x7fde9806eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.515+0000 7fde97fff640 10 monclient: _finish_hunting 0 2023-12-11T20:17:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.515+0000 7fde97fff640 1 monclient: found mon.noname-c 2023-12-11T20:17:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.515+0000 7fde97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.515+0000 7fde97fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:17:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.515+0000 7fde97fff640 1 -- 172.21.15.86:0/2017795029 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde98070f80 con 0x7fde9813e2f0 2023-12-11T20:17:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.518+0000 7fde97fff640 1 --2- 172.21.15.86:0/2017795029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde9813e2f0 0x7fde9813b5a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde840097a0 tx=0x7fde8402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=456ad9013def8b17 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:05.518 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.520+0000 7fde977fe640 1 -- 172.21.15.86:0/2017795029 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8402e070 con 0x7fde9813e2f0 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.520+0000 7fde977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.520+0000 7fde977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.520+0000 7fde977fe640 10 monclient: dump: 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:05.519 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:05.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.521+0000 7fde977fe640 10 monclient: _finish_auth 0 2023-12-11T20:17:05.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.521+0000 7fde977fe640 10 monclient: _check_auth_tickets 2023-12-11T20:17:05.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.521+0000 7fde977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:05.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.521+0000 7fde9f04b640 5 monclient: authenticate success, global_id 4453 2023-12-11T20:17:05.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.521+0000 7fde9f04b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.522+0000 7fde977fe640 1 -- 172.21.15.86:0/2017795029 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde840315a0 con 0x7fde9813e2f0 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.522+0000 7fde977fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.522+0000 7fde977fe640 1 -- 172.21.15.86:0/2017795029 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84031910 con 0x7fde9813e2f0 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.522+0000 7fde977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.522+0000 7fde977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.522+0000 7fde977fe640 10 monclient: dump: 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.522+0000 7fde9f04b640 10 monclient: get_monmap_and_config success 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.522+0000 7fde9f04b640 4 set_mon_vals no callback set 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 10 monclient: shutdown 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 1 -- 172.21.15.86:0/2017795029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde9813e2f0 msgr2=0x7fde9813b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 1 --2- 172.21.15.86:0/2017795029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde9813e2f0 0x7fde9813b5a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde840097a0 tx=0x7fde8402b840 comp rx=0 tx=0).stop 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 1 -- 172.21.15.86:0/2017795029 shutdown_connections 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 1 --2- 172.21.15.86:0/2017795029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde9813e2f0 0x7fde9813b5a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 1 -- 172.21.15.86:0/2017795029 >> 172.21.15.86:0/2017795029 conn(0x7fde980b3590 msgr2=0x7fde980b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 1 -- 172.21.15.86:0/2017795029 shutdown_connections 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 1 -- 172.21.15.86:0/2017795029 wait complete. 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 10 monclient: build_initial_monmap 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.523+0000 7fde9f04b640 10 monclient: monmap: 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 librados: starting msgr at 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 librados: starting objecter 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 Processor -- start 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 -- start start 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 librados: setting wanted keys 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 librados: calling monclient init 2023-12-11T20:17:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient: init 2023-12-11T20:17:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient: _reopen_session rank -1 2023-12-11T20:17:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:17:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde980719d0 0x7fde98146230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient(hunting): picked mon.noname-b con 0x7fde980719d0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9813e2f0 0x7fde980accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient(hunting): picked mon.noname-c con 0x7fde9813e2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde98146bf0 0x7fde980ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient(hunting): picked mon.noname-a con 0x7fde98146bf0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde980aa6a0 con 0x7fde98146bf0 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9814bad0 con 0x7fde980719d0 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde980ac5d0 con 0x7fde9813e2f0 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient(hunting): _renew_subs 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.524+0000 7fde9f04b640 10 monclient(hunting): authenticate will time out at 1511.853966s 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde980719d0 0x7fde98146230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9d5e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde98146bf0 0x7fde980ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde980719d0 0x7fde98146230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:52630/0 (socket says 172.21.15.86:52630) 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 1 -- 172.21.15.86:0/3601333479 learned_addr learned my addr 172.21.15.86:0/3601333479 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde97fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9813e2f0 0x7fde980accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 10 monclient(hunting): get_auth_request con 0x7fde980719d0 auth_method 0 2023-12-11T20:17:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9d5e8640 10 monclient(hunting): get_auth_request con 0x7fde98146bf0 auth_method 0 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9d5e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9d5e8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9d5e8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde97fff640 10 monclient(hunting): get_auth_request con 0x7fde9813e2f0 auth_method 0 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde97fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde97fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.525+0000 7fde9cde7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.526+0000 7fde9d5e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.526+0000 7fde9d5e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.526+0000 7fde9d5e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.526+0000 7fde97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.526+0000 7fde97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.526+0000 7fde97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 10 monclient(hunting): handle_auth_done global_id 4456 payload 824 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 1 -- 172.21.15.86:0/3601333479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9813e2f0 msgr2=0x7fde980accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9813e2f0 0x7fde980accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 1 -- 172.21.15.86:0/3601333479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde98146bf0 msgr2=0x7fde980ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde98146bf0 0x7fde980ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 10 monclient: _finish_hunting 0 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 1 monclient: found mon.noname-b 2023-12-11T20:17:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.528+0000 7fde9cde7640 1 -- 172.21.15.86:0/3601333479 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde84009400 con 0x7fde980719d0 2023-12-11T20:17:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.530+0000 7fde9cde7640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde980719d0 0x7fde98146230 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00c500 tx=0x7fde8c00c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.530+0000 7fde957fa640 10 client.?.objecter ms_handle_connect 0x7fde980719d0 2023-12-11T20:17:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.530+0000 7fde957fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:05.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde957fa640 1 -- 172.21.15.86:0/3601333479 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c005ce0 con 0x7fde980719d0 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde957fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde957fa640 10 monclient: dump: 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde957fa640 10 monclient: _finish_auth 0 2023-12-11T20:17:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde957fa640 10 monclient: _check_auth_tickets 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde957fa640 1 -- 172.21.15.86:0/3601333479 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c005e40 con 0x7fde980719d0 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.533+0000 7fde9f04b640 5 monclient: authenticate success, global_id 4456 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde9f04b640 10 monclient: _renew_subs 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde9f04b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde980ac8b0 con 0x7fde980719d0 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde957fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde957fa640 1 -- 172.21.15.86:0/3601333479 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c007330 con 0x7fde980719d0 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde957fa640 10 monclient: dump: 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde96ffd640 4 set_mon_vals no callback set 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde9f04b640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde9f04b640 10 monclient: _renew_subs 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde9f04b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde980aab80 con 0x7fde980719d0 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.534+0000 7fde9f04b640 1 librados: init done 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.535+0000 7fde9f04b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.535+0000 7fde9f04b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.535+0000 7fde9f04b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:05.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.535+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde980703c0 con 0x7fde980719d0 2023-12-11T20:17:05.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.778+0000 7fde957fa640 1 -- 172.21.15.86:0/3601333479 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fde8c007550 con 0x7fde980719d0 2023-12-11T20:17:05.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.778+0000 7fde957fa640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fde7c06c870 0x7fde7c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:05.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.778+0000 7fde97fff640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fde7c06c870 0x7fde7c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:05.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.778+0000 7fde97fff640 10 monclient: get_auth_request con 0x7fde7c06c870 auth_method 0 2023-12-11T20:17:05.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.779+0000 7fde957fa640 10 client.4456.objecter ms_handle_connect 0x7fde7c06c870 2023-12-11T20:17:05.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.779+0000 7fde97fff640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fde7c06c870 0x7fde7c06ed30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fde84036000 tx=0x7fde84008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:05.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.781+0000 7fde957fa640 1 -- 172.21.15.86:0/3601333479 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fde8c057f90 con 0x7fde980719d0 2023-12-11T20:17:05.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.781+0000 7fde957fa640 10 client.4456.objecter ms_dispatch 0x7fde98000b90 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:05.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.781+0000 7fde957fa640 3 client.4456.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:05.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.781+0000 7fde957fa640 3 client.4456.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:05.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.781+0000 7fde957fa640 20 client.4456.objecter dump_active .. 0 homeless 2023-12-11T20:17:05.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.792+0000 7fde957fa640 1 -- 172.21.15.86:0/3601333479 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde8c0542d0 con 0x7fde980719d0 2023-12-11T20:17:05.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.792+0000 7fde957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:17:05.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.792+0000 7fde957fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:17:05.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.996+0000 7fde9f04b640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-12-11T20:17:05.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.996+0000 7fde9f04b640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-12-11T20:17:05.997 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.996+0000 7fde9f04b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:17:05.997 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:05.996+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fde9804e840 con 0x7fde980719d0 2023-12-11T20:17:06.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.003+0000 7fde957fa640 1 -- 172.21.15.86:0/3601333479 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fde8c0578f0 con 0x7fde980719d0 2023-12-11T20:17:06.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.003+0000 7fde957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-12-11T20:17:06.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.003+0000 7fde957fa640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:17:06.002 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:17:06.002 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-11T20:17:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.005+0000 7fde9f04b640 10 librados: watch_flush enter 2023-12-11T20:17:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.005+0000 7fde9f04b640 10 librados: watch_flush exit 2023-12-11T20:17:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.005+0000 7fde9f04b640 20 client.4456.objecter shutdown clearing up homeless session... 2023-12-11T20:17:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.005+0000 7fde9f04b640 10 client.4456.objecter successfully canceled tick 2023-12-11T20:17:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fde7c06c870 msgr2=0x7fde7c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fde7c06c870 0x7fde7c06ed30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fde84036000 tx=0x7fde84008030 comp rx=0 tx=0).stop 2023-12-11T20:17:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 10 monclient: shutdown 2023-12-11T20:17:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde980719d0 msgr2=0x7fde98146230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde980719d0 0x7fde98146230 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00c500 tx=0x7fde8c00c9c0 comp rx=0 tx=0).stop 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 shutdown_connections 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fde7c06c870 0x7fde7c06ed30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9813e2f0 0x7fde980accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde98146bf0 0x7fde980ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 --2- 172.21.15.86:0/3601333479 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde980719d0 0x7fde98146230 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 >> 172.21.15.86:0/3601333479 conn(0x7fde980b3590 msgr2=0x7fde9813c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 shutdown_connections 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 -- 172.21.15.86:0/3601333479 wait complete. 2023-12-11T20:17:06.006 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:17:06.006+0000 7fde9f04b640 1 librados: shutdown 2023-12-11T20:17:06.015 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-11T20:17:06.015 INFO:teuthology.run_tasks:Running task workunit... 2023-12-11T20:17:06.032 INFO:tasks.workunit:Pulling workunits from ref c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-11T20:17:06.032 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-11T20:17:06.033 DEBUG:teuthology.orchestra.run.smithi086:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-11T20:17:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T20:17:06.047 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-11T20:17:06.048 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-11T20:17:06.102 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-11T20:17:06.102 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-11T20:17:06.157 INFO:tasks.workunit:timeout=6h 2023-12-11T20:17:06.157 INFO:tasks.workunit:cleanup=True 2023-12-11T20:17:06.158 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-11T20:17:06.213 INFO:tasks.workunit.client.0.smithi086.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-11T20:17:16.782 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:29.890 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:16.787 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:29.896 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:16.830 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:29.898 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:16.834 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:29.903 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:16.835 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:29.944 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:16.849 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:29.958 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:16.855 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:29.964 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:16.876 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:29.944 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:16.899 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:30.007 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:36.822 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:49.930 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:36.903 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:50.011 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:36.925 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:50.033 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:36.928 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:01:50.037 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:36.951 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:50.020 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:36.952 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:50.021 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:36.972 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:50.040 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:36.982 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:50.090 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:37.007 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:01:50.116 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:17:54.911 INFO:tasks.workunit.client.0.smithi086.stderr:Note: switching to 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'. 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr: 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr:state without impacting any branches by switching back to a branch. 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr: 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr: 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr: git switch -c 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr: 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr:Or undo this operation with: 2023-12-11T20:17:54.912 INFO:tasks.workunit.client.0.smithi086.stderr: 2023-12-11T20:17:54.913 INFO:tasks.workunit.client.0.smithi086.stderr: git switch - 2023-12-11T20:17:54.913 INFO:tasks.workunit.client.0.smithi086.stderr: 2023-12-11T20:17:54.913 INFO:tasks.workunit.client.0.smithi086.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-11T20:17:54.913 INFO:tasks.workunit.client.0.smithi086.stderr: 2023-12-11T20:17:54.913 INFO:tasks.workunit.client.0.smithi086.stderr:HEAD is now at c56ad3eaf07 Merge branch 'wip-neorados-learning-from-experience' of https://github.com/adamemerson/ceph into wip-yuri3-testing-2023-12-07-0727 2023-12-11T20:17:54.916 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-11T20:17:54.978 INFO:tasks.workunit.client.0.smithi086.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-11T20:17:54.980 INFO:tasks.workunit.client.0.smithi086.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-11T20:17:54.980 INFO:tasks.workunit.client.0.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-11T20:17:55.584 INFO:tasks.workunit.client.0.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-11T20:17:55.650 INFO:tasks.workunit.client.0.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-11T20:17:55.691 INFO:tasks.workunit.client.0.smithi086.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-11T20:17:55.695 INFO:tasks.workunit.client.0.smithi086.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-11T20:17:55.695 INFO:tasks.workunit.client.0.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-11T20:17:55.731 INFO:tasks.workunit.client.0.smithi086.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-11T20:17:55.734 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-11T20:17:55.734 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-11T20:17:55.792 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-12-11T20:17:55.793 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-12-11T20:17:55.793 DEBUG:teuthology.orchestra.run.smithi086:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c56ad3eaf0784383bf22c04e35d26e15c4baeafe TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ parallel=1 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' '' = --serial ']' 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ color= 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -t 1 ']' 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ declare -A pids 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_aio 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_aio' 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_aio 2023-12-11T20:17:55.852 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.853 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_aio 2023-12-11T20:17:55.853 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92225 2023-12-11T20:17:55.853 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_aio on pid 92225' 2023-12-11T20:17:55.853 INFO:tasks.workunit.client.0.smithi086.stdout:test api_aio on pid 92225 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92225 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_aio_pp 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_aio_pp' 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_aio_pp 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.856 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.857 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_aio_pp 2023-12-11T20:17:55.858 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92232 2023-12-11T20:17:55.859 INFO:tasks.workunit.client.0.smithi086.stdout:test api_aio_pp on pid 92232 2023-12-11T20:17:55.863 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_aio_pp on pid 92232' 2023-12-11T20:17:55.863 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92232 2023-12-11T20:17:55.863 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.863 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_io 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_io' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_io 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.865 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.866 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_io 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92241 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_io on pid 92241' 2023-12-11T20:17:55.867 INFO:tasks.workunit.client.0.smithi086.stdout:test api_io on pid 92241 2023-12-11T20:17:55.872 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92241 2023-12-11T20:17:55.872 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.872 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-12-11T20:17:55.872 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.872 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_io_pp 2023-12-11T20:17:55.872 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_io_pp' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_io_pp 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:55.873 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.874 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.875 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_io_pp 2023-12-11T20:17:55.876 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92254 2023-12-11T20:17:55.877 INFO:tasks.workunit.client.0.smithi086.stdout:test api_io_pp on pid 92254 2023-12-11T20:17:55.883 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_io_pp on pid 92254' 2023-12-11T20:17:55.883 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92254 2023-12-11T20:17:55.883 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.883 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.883 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-12-11T20:17:55.883 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_asio 2023-12-11T20:17:55.883 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_asio' 2023-12-11T20:17:55.883 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:55.884 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:55.884 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:55.884 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:55.884 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.884 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:55.884 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_asio 2023-12-11T20:17:55.884 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:55.885 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.885 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:55.885 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:55.885 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.885 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:55.886 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:55.886 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.886 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.886 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:55.886 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.886 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.887 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.888 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:55.889 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.890 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.890 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.890 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:55.890 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.890 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.890 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.890 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:55.890 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.891 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_asio 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:55.892 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:55.893 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.893 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:55.893 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.893 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:55.893 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92269 2023-12-11T20:17:55.893 INFO:tasks.workunit.client.0.smithi086.stdout:test api_asio on pid 92269 2023-12-11T20:17:55.899 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_asio on pid 92269' 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92269 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:55.900 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_list 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_list' 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_list 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.901 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.902 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92225/exe 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.905 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.906 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:55.907 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_list 2023-12-11T20:17:55.908 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92285 2023-12-11T20:17:55.908 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_list on pid 92285' 2023-12-11T20:17:55.908 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-12-11T20:17:55.908 INFO:tasks.workunit.client.0.smithi086.stdout:test api_list on pid 92285 2023-12-11T20:17:55.916 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92285 2023-12-11T20:17:55.916 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.916 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.916 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:55.916 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:55.916 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:55.916 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_lock 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_lock' 2023-12-11T20:17:55.917 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:55.918 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_lock 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92232/exe 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_aio: /' 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.921 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.922 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_aio.log 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.923 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:55.924 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.924 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.924 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.924 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:55.924 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.924 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:55.924 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_lock 2023-12-11T20:17:55.926 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:55.926 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:55.926 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:55.926 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:55.927 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:55.927 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92304 2023-12-11T20:17:55.927 INFO:tasks.workunit.client.0.smithi086.stdout:test api_lock on pid 92304 2023-12-11T20:17:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_lock on pid 92304' 2023-12-11T20:17:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92304 2023-12-11T20:17:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_lock_pp 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_lock_pp' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.941 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_aio_pp: /' 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_lock_pp 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:55.943 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.944 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:55.945 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.945 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.945 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:55.945 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.945 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:55.945 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:55.945 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.945 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.946 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.946 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.946 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.946 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.946 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:55.946 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.946 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:55.946 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.947 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.948 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:55.949 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.949 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:55.949 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:55.949 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.949 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:55.949 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.949 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:55.950 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:55.950 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:55.950 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.950 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:55.950 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.950 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:55.951 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:55.951 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:55.951 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.951 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:55.951 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.951 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:55.951 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.951 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:55.952 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.952 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92241/exe 2023-12-11T20:17:55.952 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.952 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:55.952 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:55.952 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:55.952 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:55.952 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_lock_pp 2023-12-11T20:17:55.953 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92325 2023-12-11T20:17:55.953 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_lock_pp on pid 92325' 2023-12-11T20:17:55.953 INFO:tasks.workunit.client.0.smithi086.stdout:test api_lock_pp on pid 92325 2023-12-11T20:17:55.968 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92325 2023-12-11T20:17:55.968 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.968 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:55.969 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:55.970 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_io.log 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:55.971 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_io: /' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.972 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.973 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_misc 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_misc' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.974 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_misc 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.975 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92254/exe 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:55.976 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:55.977 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:55.978 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_misc 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:55.979 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:55.980 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.980 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:55.980 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92352 2023-12-11T20:17:55.980 INFO:tasks.workunit.client.0.smithi086.stdout:test api_misc on pid 92352 2023-12-11T20:17:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_misc on pid 92352' 2023-12-11T20:17:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92352 2023-12-11T20:17:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_io_pp: /' 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:55.994 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_misc_pp 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.995 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_misc_pp' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92269/exe 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:55.996 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.997 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_misc_pp 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.000 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.001 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92285/exe 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_asio.log 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.002 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_list.log 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_misc_pp 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92379 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_misc_pp on pid 92379' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.006 INFO:tasks.workunit.client.0.smithi086.stdout:test api_misc_pp on pid 92379 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92379 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_tier_pp 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.010 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_tier_pp' 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_tier_pp 2023-12-11T20:17:56.011 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.012 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.013 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.013 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_tier_pp 2023-12-11T20:17:56.013 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92389 2023-12-11T20:17:56.013 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_tier_pp on pid 92389' 2023-12-11T20:17:56.013 INFO:tasks.workunit.client.0.smithi086.stdout:test api_tier_pp on pid 92389 2023-12-11T20:17:56.016 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92389 2023-12-11T20:17:56.016 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_asio: /' 2023-12-11T20:17:56.016 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_pool 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_pool' 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.017 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92304/exe 2023-12-11T20:17:56.018 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.019 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.019 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.019 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_pool 2023-12-11T20:17:56.019 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_pool 2023-12-11T20:17:56.019 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92400 2023-12-11T20:17:56.019 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_pool on pid 92400' 2023-12-11T20:17:56.019 INFO:tasks.workunit.client.0.smithi086.stdout:test api_pool on pid 92400 2023-12-11T20:17:56.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92400 2023-12-11T20:17:56.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_list: /' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.030 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.031 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.032 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_lock.log 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_snapshots 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_snapshots' 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_snapshots 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.034 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_lock: /' 2023-12-11T20:17:56.035 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.036 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.037 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.038 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.038 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.038 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.038 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2023-12-11T20:17:56.038 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.038 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_snapshots 2023-12-11T20:17:56.038 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92432 2023-12-11T20:17:56.038 INFO:tasks.workunit.client.0.smithi086.stdout:test api_snapshots on pid 92432 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_snapshots on pid 92432' 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92432 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.059 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.060 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.061 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.062 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.063 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.064 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.065 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92325/exe 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.066 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_snapshots_pp 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92352/exe 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_snapshots_pp' 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.067 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.068 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.069 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.070 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.071 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_snapshots_pp 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.072 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_misc: /' 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_lock_pp: /' 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.073 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.074 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.075 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.076 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92389/exe 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.077 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.078 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_snapshots_pp 2023-12-11T20:17:56.078 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92502 2023-12-11T20:17:56.078 INFO:tasks.workunit.client.0.smithi086.stdout:test api_snapshots_pp on pid 92502 2023-12-11T20:17:56.083 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_snapshots_pp on pid 92502' 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92502 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_misc.log 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_stat 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_stat' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.086 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_tier_pp: /' 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_stat 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.087 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [----------] Global test environment set-up. 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [----------] 12 tests from AsioRados 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.948+0000 7f50b9045780 10 monclient: get_monmap_and_config 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.948+0000 7f50b9045780 10 monclient: build_initial_monmap 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.948+0000 7f50b9045780 10 monclient: monmap: 2023-12-11T20:17:56.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: epoch 0 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: last_changed 0.000000 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: created 0.000000 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: election_strategy: 1 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.949+0000 7f50b9045780 1 Processor -- start 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.949+0000 7f50b9045780 1 -- start start 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.949+0000 7f50b9045780 10 monclient: init 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient: _reopen_session rank -1 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:17:56.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c136a0e280 0x55c136a4bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient(hunting): picked mon.noname-c con 0x55c136a0e280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 0x55c136a51330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient(hunting): picked mon.noname-a con 0x55c136a4c5c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136b79060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient(hunting): picked mon.noname-b con 0x55c136a4cb70 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1369c8780 con 0x55c136a4c5c0 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c136962970 con 0x55c136a4cb70 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c136962610 con 0x55c136a0e280 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient(hunting): _renew_subs 2023-12-11T20:17:56.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.950+0000 7f50b9045780 10 monclient(hunting): authenticate will time out at 1562.279672s 2023-12-11T20:17:56.091 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_stat 2023-12-11T20:17:56.091 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92536 2023-12-11T20:17:56.091 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_stat on pid 92536' 2023-12-11T20:17:56.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: test api_stat on pid 92536 2023-12-11T20:17:56.103 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92536 2023-12-11T20:17:56.103 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.103 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.104 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_stat_pp 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_stat_pp' 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.105 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-12-11T20:17:56.106 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.107 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92379/exe 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_stat_pp 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.110 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92432/exe 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.111 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92400/exe 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.112 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_stat_pp 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.113 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.114 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92573 2023-12-11T20:17:56.114 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_stat_pp on pid 92573' 2023-12-11T20:17:56.114 INFO:tasks.workunit.client.0.smithi086.stdout:test api_stat_pp on pid 92573 2023-12-11T20:17:56.126 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92573 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.127 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_watch_notify 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.128 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_watch_notify' 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_misc_pp: /' 2023-12-11T20:17:56.129 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.130 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.131 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_pool.log 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_pool: /' 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.133 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_watch_notify 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_watch_notify 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92624 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_watch_notify on pid 92624' 2023-12-11T20:17:56.137 INFO:tasks.workunit.client.0.smithi086.stdout:test api_watch_notify on pid 92624 2023-12-11T20:17:56.140 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92624 2023-12-11T20:17:56.140 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.140 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.140 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_snapshots: /' 2023-12-11T20:17:56.140 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.140 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.140 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.140 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.141 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.142 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.142 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.142 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_watch_notify_pp 2023-12-11T20:17:56.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_watch_notify_pp' 2023-12-11T20:17:56.142 INFO:tasks.workunit.client.0.smithi086.stdout:2023-12-11T20:17:55.959+0000 7f50b6b8a640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136b79060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:56.142 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.959+0000 7f50b6b8a640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136b79060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:55148/0 (socket says 172.21.15.86:55148) 2023-12-11T20:17:56.142 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.959+0000 7f50b6b8a640 1 -- 172.21.15.86:0/2253498574 learned_addr learned my addr 172.21.15.86:0/2253498574 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:56.142 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.959+0000 7f50b6b8a640 10 monclient(hunting): get_auth_request con 0x55c136a4cb70 auth_method 0 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.959+0000 7f50b6b8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.959+0000 7f50b6b8a640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.959+0000 7f50b6b8a640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.960+0000 7f50b6b8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.960+0000 7f50b6b8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.960+0000 7f50b6b8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 10 monclient(hunting): handle_auth_done global_id 4468 payload 274 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 1 -- 172.21.15.86:0/2253498574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c136a0e280 msgr2=0x55c136a4bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 1 --2- 172.21.15.86:0/2253498574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c136a0e280 0x55c136a4bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 1 -- 172.21.15.86:0/2253498574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 msgr2=0x55c136a51330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 1 --2- 172.21.15.86:0/2253498574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 0x55c136a51330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 10 monclient: _finish_hunting 0 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 1 monclient: found mon.noname-b 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:56.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.962+0000 7f50b6b8a640 1 -- 172.21.15.86:0/2253498574 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a8005de0 con 0x55c136a4cb70 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:55.970+0000 7f50b6b8a640 1 --2- 172.21.15.86:0/2253498574 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136b79060 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009200 tx=0x7f50b0005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dc7e07de3a9cbcf server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 1 -- 172.21.15.86:0/2253498574 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0009cb0 con 0x55c136a4cb70 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: dump: 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: epoch 1 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: min_mon_release 18 (reef) 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: election_strategy: 1 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:56.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: _finish_auth 0 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: _check_auth_tickets 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 1 -- 172.21.15.86:0/2253498574 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b000f040 con 0x55c136a4cb70 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 1 -- 172.21.15.86:0/2253498574 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b000da20 con 0x55c136a4cb70 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b4b86640 10 monclient: dump: 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: epoch 1 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:56.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: min_mon_release 18 (reef) 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: election_strategy: 1 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b9045780 5 monclient: authenticate success, global_id 4468 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b9045780 10 monclient: get_monmap_and_config success 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.008+0000 7f50b9045780 4 set_mon_vals no callback set 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 10 monclient: shutdown 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 -- 172.21.15.86:0/2253498574 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 msgr2=0x55c136b79060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:56.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 --2- 172.21.15.86:0/2253498574 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136b79060 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009200 tx=0x7f50b0005d80 comp rx=0 tx=0).stop 2023-12-11T20:17:56.147 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92502/exe 2023-12-11T20:17:56.147 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.147 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.147 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 -- 172.21.15.86:0/2253498574 shutdown_connections 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 --2- 172.21.15.86:0/2253498574 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136b79060 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 --2- 172.21.15.86:0/2253498574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 0x55c136a51330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 --2- 172.21.15.86:0/2253498574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c136a0e280 0x55c136a4bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 -- 172.21.15.86:0/2253498574 >> 172.21.15.86:0/2253498574 conn(0x55c136a4f090 msgr2=0x55c136adedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 -- 172.21.15.86:0/2253498574 shutdown_connections 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 1 -- 172.21.15.86:0/2253498574 wait complete. 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 10 monclient: build_initial_monmap 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.009+0000 7f50b9045780 10 monclient: monmap: 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: epoch 0 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: last_changed 0.000000 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: created 0.000000 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-11T20:17:56.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: election_strategy: 1 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.010+0000 7f50b9045780 1 librados: starting msgr at 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.010+0000 7f50b9045780 1 librados: starting objecter 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.010+0000 7f50b9045780 1 Processor -- start 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.010+0000 7f50b9045780 1 -- start start 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.010+0000 7f50b9045780 1 librados: setting wanted keys 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.010+0000 7f50b9045780 1 librados: calling monclient init 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.010+0000 7f50b9045780 10 monclient: init 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient: _reopen_session rank -1 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c136a0e280 0x55c136ad6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:56.149 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient(hunting): picked mon.noname-c con 0x55c136a0e280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:17:56.150 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 0x55c136ad7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:56.150 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient(hunting): picked mon.noname-a con 0x55c136a4c5c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-12-11T20:17:56.156 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.157 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.158 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.159 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136a10d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient(hunting): picked mon.noname-b con 0x55c136a4cb70 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1369424e0 con 0x55c136a4c5c0 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c136a0fec0 con 0x55c136a4cb70 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient(hunting): start opening mon connection 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c136ad7870 con 0x55c136a0e280 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient(hunting): _renew_subs 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.011+0000 7f50b9045780 10 monclient(hunting): authenticate will time out at 1562.340399s 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.018+0000 7f50b5b88640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 0x55c136ad7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.019+0000 7f50b5b88640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 0x55c136ad7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40988/0 (socket says 172.21.15.86:40988) 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.019+0000 7f50b5b88640 1 -- 172.21.15.86:0/1257650968 learned_addr learned my addr 172.21.15.86:0/1257650968 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.019+0000 7f50b5b88640 10 monclient(hunting): get_auth_request con 0x55c136a4c5c0 auth_method 0 2023-12-11T20:17:56.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.019+0000 7f50b5b88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.019+0000 7f50b5b88640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.019+0000 7f50b5b88640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.020+0000 7f50b5b88640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.020+0000 7f50b5b88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.020+0000 7f50b5b88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.023+0000 7f50b6b8a640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136a10d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.024+0000 7f50b6b8a640 10 monclient(hunting): get_auth_request con 0x55c136a4cb70 auth_method 0 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.024+0000 7f50b6b8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.024+0000 7f50b6b8a640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:17:56.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.024+0000 7f50b6b8a640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:17:56.166 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.166 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_watch_notify_pp 2023-12-11T20:17:56.166 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92536/exe 2023-12-11T20:17:56.166 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.166 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.166 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.167 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.168 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.169 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.170 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_watch_notify_pp 2023-12-11T20:17:56.170 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92684 2023-12-11T20:17:56.170 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000test api_watch_notify_pp on pid 92684 2023-12-11T20:17:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_watch_notify_pp on pid 92684' 2023-12-11T20:17:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92684 2023-12-11T20:17:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-12-11T20:17:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_stat: /' 2023-12-11T20:17:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_stat.log 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_cmd 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_cmd' 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_cmd 2023-12-11T20:17:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_cmd 2023-12-11T20:17:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92711 2023-12-11T20:17:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_cmd on pid 92711' 2023-12-11T20:17:56.183 INFO:tasks.workunit.client.0.smithi086.stdout:test api_cmd on pid 92711 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92711 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92573/exe 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.190 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_cmd_pp 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_cmd_pp' 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.191 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.192 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.193 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.194 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_stat_pp: /' 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.195 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_cmd_pp 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.196 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.197 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.197 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.197 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.197 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.197 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_cmd_pp 2023-12-11T20:17:56.197 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92734 2023-12-11T20:17:56.197 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_cmd_pp on pid 92734' 2023-12-11T20:17:56.197 INFO:tasks.workunit.client.0.smithi086.stdout:test api_cmd_pp on pid 92734 2023-12-11T20:17:56.203 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92734 2023-12-11T20:17:56.203 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.203 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-12-11T20:17:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_service 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_service' 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_service 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.205 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.206 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.207 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.208 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.209 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.209 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.209 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_service 2023-12-11T20:17:56.209 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.209 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92747 2023-12-11T20:17:56.209 INFO:tasks.workunit.client.0.smithi086.stdout:test api_service on pid 92747 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_service on pid 92747' 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92747 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.231 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_service_pp 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_service_pp' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92624/exe 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.232 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.234 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_watch_notify: /' 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.238 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_service_pp 2023-12-11T20:17:56.239 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.240 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92711/exe 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.244 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.245 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92684/exe 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_cmd.log 2023-12-11T20:17:56.246 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.247 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.248 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_service_pp 2023-12-11T20:17:56.249 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.250 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.250 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.250 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.250 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92815 2023-12-11T20:17:56.250 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_service_pp on pid 92815' 2023-12-11T20:17:56.250 INFO:tasks.workunit.client.0.smithi086.stdout:test api_service_pp on pid 92815 2023-12-11T20:17:56.258 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92815 2023-12-11T20:17:56.258 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.258 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.258 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2023-12-11T20:17:56.258 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_cmd: /' 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_c_write_operations 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_c_write_operations' 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_c_write_operations 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.259 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.260 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92747/exe 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.261 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.262 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_service.log 2023-12-11T20:17:56.263 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_service: /' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_c_write_operations 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.264 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92855 2023-12-11T20:17:56.265 INFO:tasks.workunit.client.0.smithi086.stdout:test api_c_write_operations on pid 92855 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_c_write_operations on pid 92855' 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92855 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.269 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_c_read_operations 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_c_read_operations' 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_c_read_operations 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.270 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92734/exe 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.271 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_c_read_operations 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92865 2023-12-11T20:17:56.272 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_c_read_operations on pid 92865' 2023-12-11T20:17:56.273 INFO:tasks.workunit.client.0.smithi086.stdout:test api_c_read_operations on pid 92865 2023-12-11T20:17:56.287 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92865 2023-12-11T20:17:56.287 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.287 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.287 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.288 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.289 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.290 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.291 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.292 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s api_cls_remote_reads 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' api_cls_remote_reads' 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.293 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.294 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.295 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.296 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92815/exe 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_service_pp: /' 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo api_cls_remote_reads 2023-12-11T20:17:56.299 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.300 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-12-11T20:17:56.300 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-12-11T20:17:56.300 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=api_cls_remote_reads 2023-12-11T20:17:56.300 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92909 2023-12-11T20:17:56.300 INFO:tasks.workunit.client.0.smithi086.stdout:test api_cls_remote_reads on pid 92909 2023-12-11T20:17:56.301 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test api_cls_remote_reads on pid 92909' 2023-12-11T20:17:56.301 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92909 2023-12-11T20:17:56.301 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.301 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.301 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.301 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.301 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stdout: 7f50b5b88640 10 monclient(hunting): handle_auth_done global_id 4407 payload 824 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 1 -- 172.21.15.86:0/1257650968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c136a0e280 msgr2=0x55c136ad6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c136a0e280 0x55c136ad6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 1 -- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 msgr2=0x55c136a10d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c136a4cb70 0x55c136a10d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 10 monclient: _finish_hunting 0 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 1 monclient: found mon.noname-a 2023-12-11T20:17:56.302 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.025+0000 7f50b5b88640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0009e10 con 0x55c136a4c5c0 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.068+0000 7f50b5b88640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 0x55c136ad7330 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f50a8027050 tx=0x7f50a80113d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.068+0000 7f50a77fe640 10 client.?.objecter ms_handle_connect 0x55c136a4c5c0 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.068+0000 7f50a77fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8055640 con 0x55c136a4c5c0 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: dump: 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: epoch 1 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: min_mon_release 18 (reef) 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: election_strategy: 1 2023-12-11T20:17:56.303 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:56.304 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:56.304 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:56.304 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.304 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: _finish_auth 0 2023-12-11T20:17:56.304 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: _check_auth_tickets 2023-12-11T20:17:56.304 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:17:56.311 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.311 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.311 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.311 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.311 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.311 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.311 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.311 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92865/exe 2023-12-11T20:17:56.312 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s list_parallel 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' list_parallel' 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo list_parallel 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.313 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.314 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=list_parallel 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.317 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92948 2023-12-11T20:17:56.317 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-12-11T20:17:56.317 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test list_parallel on pid 92948' 2023-12-11T20:17:56.317 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300test list_parallel on pid 92948 2023-12-11T20:17:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92948 2023-12-11T20:17:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s open_pools_parallel 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' open_pools_parallel' 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo open_pools_parallel 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.325 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.326 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.326 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.326 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.326 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=open_pools_parallel 2023-12-11T20:17:56.326 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92967 2023-12-11T20:17:56.326 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test open_pools_parallel on pid 92967' 2023-12-11T20:17:56.326 INFO:tasks.workunit.client.0.smithi086.stdout:test open_pools_parallel on pid 92967 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92967 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.333 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.334 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.335 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s delete_pools_parallel 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92855/exe 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' delete_pools_parallel' 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.336 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo delete_pools_parallel 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.337 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=delete_pools_parallel 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92987 2023-12-11T20:17:56.338 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test delete_pools_parallel on pid 92987' 2023-12-11T20:17:56.339 INFO:tasks.workunit.client.0.smithi086.stdout:test delete_pools_parallel on pid 92987 2023-12-11T20:17:56.343 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92987 2023-12-11T20:17:56.343 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s cls 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' cls' 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo cls 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.344 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.345 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.346 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=cls 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92998 2023-12-11T20:17:56.347 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test cls on pid 92998' 2023-12-11T20:17:56.348 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.348 INFO:tasks.workunit.client.0.smithi086.stdout:test cls on pid 92998 2023-12-11T20:17:56.359 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=92998 2023-12-11T20:17:56.359 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.359 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.359 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.359 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.361 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.362 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s cmd 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.367 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' cmd' 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92948/exe 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo cmd 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=cmd 2023-12-11T20:17:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93032 2023-12-11T20:17:56.370 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test cmd on pid 93032' 2023-12-11T20:17:56.370 INFO:tasks.workunit.client.0.smithi086.stdout:test cmd on pid 93032 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93032 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_list_parallel.log 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.375 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.376 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s handler_error 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' handler_error' 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo handler_error 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ list_parallel: /' 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.377 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=handler_error 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93045 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.378 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.379 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.379 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.379 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.379 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.379 INFO:tasks.workunit.client.0.smithi086.stdout:test handler_error on pid 93045 2023-12-11T20:17:56.397 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test handler_error on pid 93045' 2023-12-11T20:17:56.397 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93045 2023-12-11T20:17:56.397 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.397 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.398 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s io 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92909/exe 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' io' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.400 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.401 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.402 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.403 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.404 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo io 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.407 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.408 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.409 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92967/exe 2023-12-11T20:17:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=io 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93100 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test io on pid 93100' 2023-12-11T20:17:56.411 INFO:tasks.workunit.client.0.smithi086.stdout:test io on pid 93100 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93100 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s list 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' list' 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.432 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2023-12-11T20:17:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.435 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.436 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.437 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.438 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.439 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo list 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.440 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.441 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92987/exe 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=list 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93178 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2023-12-11T20:17:56.449 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test list on pid 93178' 2023-12-11T20:17:56.450 INFO:tasks.workunit.client.0.smithi086.stdout:test list on pid 93178 2023-12-11T20:17:56.460 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93178 2023-12-11T20:17:56.460 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.460 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s misc 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' misc' 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/92998/exe 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93100/exe 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.464 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_io.log 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_io 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93032/exe 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ cmd: /' 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_cls 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_cls.log 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo misc 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ cls: /' 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ io: /' 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=misc 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93220 2023-12-11T20:17:56.469 INFO:tasks.workunit.client.0.smithi086.stdout:test misc on pid 93220 2023-12-11T20:17:56.471 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test misc on pid 93220' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93220 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stdout:/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a80557a0 con 0x55c136a4c5c0 2023-12-11T20:17:56.473 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8055b50 con 0x55c136a4c5c0 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50a77fe640 10 monclient: dump: 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: epoch 1 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: min_mon_release 18 (reef) 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: election_strategy: 1 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:17:56.474 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b4b86640 4 set_mon_vals no callback set 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 5 monclient: authenticate success, global_id 4407 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 10 monclient: _renew_subs 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c13696e770 con 0x55c136a4c5c0 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 10 monclient: _renew_subs 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c136961920 con 0x55c136a4c5c0 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 1 librados: init done 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.141+0000 7f50b9045780 10 librados: wait_for_osdmap waiting 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.272+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f50a8055cb0 con 0x55c136a4c5c0 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.273+0000 7f50a77fe640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f509806aa90 0x7f509806cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.273+0000 7f50b6389640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f509806aa90 0x7f509806cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:56.475 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.273+0000 7f50b6389640 10 monclient: get_auth_request con 0x7f509806aa90 auth_method 0 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_cmd 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_cmd.log 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s pool 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' pool' 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo pool 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93045/exe 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_handler_error.log 2023-12-11T20:17:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_handler_error 2023-12-11T20:17:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ handler_error: /' 2023-12-11T20:17:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=pool 2023-12-11T20:17:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93288 2023-12-11T20:17:56.483 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.274+0000 7f50b6389640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f509806aa90 0x7f509806cf50 secure :-1 s=READY pgs=89 cs=0 ltest pool on pid 93288 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test pool on pid 93288' 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93288 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s read_operations 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' read_operations' 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo read_operations 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.489 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=read_operations 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93311 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test read_operations on pid 93311' 2023-12-11T20:17:56.490 INFO:tasks.workunit.client.0.smithi086.stdout:test read_operations on pid 93311 2023-12-11T20:17:56.497 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93311 2023-12-11T20:17:56.497 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s snapshots 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2023-12-11T20:17:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' snapshots' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.499 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo snapshots 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.500 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.502 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=snapshots 2023-12-11T20:17:56.503 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93326 2023-12-11T20:17:56.504 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test snapshots on pid 93326' 2023-12-11T20:17:56.504 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2023-12-11T20:17:56.504 INFO:tasks.workunit.client.0.smithi086.stdout:test snapshots on pid 93326 2023-12-11T20:17:56.513 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93326 2023-12-11T20:17:56.513 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.513 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s watch_notify 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' watch_notify' 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.514 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo watch_notify 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.515 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93178/exe 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.520 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=watch_notify 2023-12-11T20:17:56.521 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93347 2023-12-11T20:17:56.522 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test watch_notify on pid 93347' 2023-12-11T20:17:56.523 INFO:tasks.workunit.client.0.smithi086.stdout:test watch_notify on pid 93347 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93347 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_list 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:++ printf %25s write_operations 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:+ r=' write_operations' 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.534 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_list.log 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.535 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ list: /' 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.537 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93220/exe 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ awk '{print $1}' 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ echo write_operations 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.538 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.539 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stderr:+ ff=write_operations 2023-12-11T20:17:56.540 INFO:tasks.workunit.client.0.smithi086.stdout:test write_operations on pid 93380 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93380 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ echo 'test write_operations on pid 93380' 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ pids[$f]=93380 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ ret=0 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' 1 -eq 1 ']' 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92389 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92389 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2023-12-11T20:17:56.567 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ misc: /' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -f /etc/bashrc ']' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_misc.log 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+ . /etc/bashrc 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_misc 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -z '' ']' 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.569 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ BASHRCSOURCED=Y 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ shopt -q login_shell 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.570 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93288/exe 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.571 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:+++ umask 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.572 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' 0022 -eq 0 ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ SHELL=/bin/bash 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.573 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.574 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.575 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93311/exe 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.576 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'grep=grep --color=auto' 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorls.sh 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' '!' -t 0 ']' 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:+++ return 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.577 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export S_COLORS=auto 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:+++ S_COLORS=auto 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_pool 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ pool: /' 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_pool.log 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.579 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_read_operations 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_read_operations.log 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ read_operations: /' 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.580 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93326/exe 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.581 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.582 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_snapshots 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ snapshots: /' 2023-12-11T20:17:56.583 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.584 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.585 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-11T20:17:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93347/exe 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/gawk.sh 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/lang.sh 2023-12-11T20:17:56.589 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/locale 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.590 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_watch_notify 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ watch_notify: /' 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_watch_notify.log 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_snapshots.log 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:++++ export LANG=en_US.UTF-8 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:++++ LANG=en_US.UTF-8 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+++ LANG=en_US.UTF-8 2023-12-11T20:17:56.591 INFO:tasks.workunit.client.0.smithi086.stderr:+++ unset LANG_backup config 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n '' ']' 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' dumb = linux ']' 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/less.sh 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++ for i in /etc/profile.d/*.sh 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' '' ']' 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++ . /etc/profile.d/which2.sh 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:+++++ readlink /proc/93380/exe 2023-12-11T20:17:56.592 INFO:tasks.workunit.client.0.smithi086.stderr:++++ basename /usr/bin/bash 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_declare='declare -f' 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+++ which_opt=-f 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export which_declare 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+++ export -f which 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset i 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:++ unset -f pathmunge 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+ export PATH 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+ unset rc 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+ ceph_test_neorados_write_operations 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+ sed 's/^/ write_operations: /' 2023-12-11T20:17:56.593 INFO:tasks.workunit.client.0.smithi086.stderr:+ tee ceph_test_neorados_write_operations.log 2023-12-11T20:17:57.453 INFO:tasks.workunit.client.0.smithi086.stdout:=1 rev1=1 crypto rx=0x7f50ac005ee0 tx=0x7f50ac005490 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:17:57.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.274+0000 7f50a77fe640 10 client.4407.objecter ms_handle_connect 0x7f509806aa90 2023-12-11T20:17:57.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.280+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f50a8099820 con 0x55c136a4c5c0 2023-12-11T20:17:57.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.280+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(22..22 src has 1..22) v4 2023-12-11T20:17:57.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.280+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [22,22] > 0 2023-12-11T20:17:57.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.280+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding full epoch 22 2023-12-11T20:17:57.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.280+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:17:57.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.280+0000 7f50b9045780 10 librados: wait_for_osdmap done waiting 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.280+0000 7f50b9045780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.280+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55c1369c1760 con 0x55c136a4c5c0 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.342+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50a8095460 con 0x55c136a4c5c0 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.342+0000 7f50a77fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.342+0000 7f50a7fff640 10 client.4407.objecter _get_latest_version latest 22, have it 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.342+0000 7f50b9045780 10 client.4407.objecter create_pool name=ceph_test_rados_api_asio 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.342+0000 7f50b9045780 10 client.4407.objecter pool_op_submit 1 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.342+0000 7f50b9045780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:56.342+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55c1369d2810 con 0x55c136a4c5c0 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v23) v1 ==== 43+0+0 (secure 0 0 0) 0x7f50a806e020 con 0x55c136a4c5c0 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 pool_op_reply(tid 1 (0) Success v23) v1 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 10 client.4407.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v23) v1 2023-12-11T20:17:57.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 10 client.4407.objecter have request 1 at 0x55c136ad92e0 Op: create 2023-12-11T20:17:57.455 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 20 client.4407.objecter waiting for client to reach epoch 23 before calling back 2023-12-11T20:17:57.455 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:17:57.455 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 10 monclient: _renew_subs 2023-12-11T20:17:57.455 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:17:58.305 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=23}) v3 -- 0x7f50980707d0 con 0x55c136a4c5c0 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.455+0000 7f50a77fe640 10 client.4407.objecter done 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 4711+0+0 (secure 0 0 0) 0x7f50a8099120 con 0x55c136a4c5c0 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(23..23 src has 1..23) v4 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [23,23] > 22 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 23 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 10 librados: set snap read head -> 1 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 10 librados: writefull oid=exist nspace= 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136a112a0 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 23 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:17:58.306 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x55c136a11d60 0x55c136ac3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [writefull 0~5 in=5b] tid 2 osd.6 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 2 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 15 client.4407.objecter _send_op 2 to 8.1 on osd.6 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:2 8.1 8:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) v8 -- 0x55c136ac4300 con 0x55c136a11d60 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:57.578+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:58.214+0000 7f50b6b8a640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x55c136a11d60 0x55c136ac3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:58.230+0000 7f50b6b8a640 10 monclient: get_auth_request con 0x55c136a11d60 auth_method 0 2023-12-11T20:17:58.307 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:58.275+0000 7f50b6b8a640 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x55c136a11d60 0x55c136ac3dc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:18:07.734 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:02:20.842 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:07.767 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:02:20.875 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:07.857 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:20.926 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:07.869 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:20.937 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:07.906 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:02:21.015 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:07.992 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:21.060 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:07.998 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:02:21.107 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:08.011 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:02:21.120 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:08.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: api_service: [==========] Running 4 tests from 1 test suite. 2023-12-11T20:18:08.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [----------] Global test environment set-up. 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [----------] 4 tests from LibRadosService 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5477 ms) 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ OK ] LibRadosService.RegisterLate (566 ms) 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: cluster: 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: id: bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: health: HEALTH_WARN 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 22 pool(s) do not have an application enabled 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: services: 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: mgr: x(active, since 119s), standbys: y 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: osd: 8 osds: 8 up (since 76s), 8 in (since 119s) 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: data: 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: pools: 37 pools, 1129 pgs 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: pgs: 73.428% pgs unknown 2023-12-11T20:18:08.111 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 25.775% pgs not active 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 829 unknown 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 258 creating+peering 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 33 creating+activating 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 9 active+clean 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: cluster: 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: id: bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: health: HEALTH_WARN 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 16 pool(s) do not have an application enabled 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: services: 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-12-11T20:18:08.112 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: osd: 8 osds: 8 up (since 78s), 8 in (since 2m) 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: data: 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: pools: 36 pools, 1073 pgs 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: pgs: 69.338% pgs unknown 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 29.823% pgs not active 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 744 unknown 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 216 creating+peering 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 104 creating+activating 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 9 active+clean 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.113 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: cluster: 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: id: bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: health: HEALTH_WARN 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 16 pool(s) do not have an application enabled 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: services: 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: osd: 8 osds: 8 up (since 81s), 8 in (since 2m) 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: data: 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: pools: 36 pools, 1073 pgs 2023-12-11T20:18:08.114 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: objects: 4 objects, 449 KiB 2023-12-11T20:18:08.171 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:02:21.280 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:08.733 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: usage: 214 MiB used, 720 GiB / 722023-12-11T20:17:58.275+0000 7f50a77fe640 10 client.4407.objecter ms_handle_connect 0x55c136a11d60 2023-12-11T20:18:08.733 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:59.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.733 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:59.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.733 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:59.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.733 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:17:59.011+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:18:08.733 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:00.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.733 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:00.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.733 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:00.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:01.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:01.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:01.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:01.142+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:02.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:02.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:02.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:03.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:03.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:03.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.734 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:04.010+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:04.010+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:04.010+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:05.010+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:05.010+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:05.010+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:06.010+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:06.010+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:06.010+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:06.142+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:07.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:07.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:07.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:08.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:08.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:08.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.733+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 1 ==== osd_map(24..26 src has 1..26) v4 ==== 19071+0+0 (crc 0 0 0) 0x7f50b000d7e0 con 0x55c136a11d60 2023-12-11T20:18:08.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.733+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(24..26 src has 1..26) v4 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.733+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [24,26] > 23 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.733+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 24 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 24 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 25 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 25 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 26 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:11.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 26 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:08.734+0000 7f50a77fe640 20 client.4407.objecter 2 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:09.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:09.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:09.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:09.011+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:10.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:10.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:10.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.143+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:11.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.143+0000 7f50b85ed640 2 client.4407.objecter tid 2 on osd.6 is laggy 2023-12-11T20:18:11.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.143+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:11.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.143+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:18:11.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.143+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:11.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.143+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.143+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003320 con 0x55c136a11d60 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 9 ==== osd_map(27..31 src has 1..31) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f50a809d270 con 0x55c136a4c5c0 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(27..31 src has 1..31) v4 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [27,31] > 26 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 27 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 27 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 28 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:13.011 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 28 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 29 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 29 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 30 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 30 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 31 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 31 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:13.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:13.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:11.189+0000 7f50a77fe640 20 client.4407.objecter 2 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:18:13.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:12.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:13.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:12.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:13.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:12.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:13.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:13.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:13.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:13.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:16.175 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:13.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:16.175 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:14.011+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:16.175 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:14.011+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:14.011+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:15.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:15.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:15.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.144+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.144+0000 7f50b85ed640 2 client.4407.objecter tid 2 on osd.6 is laggy 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.144+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.144+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.144+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.144+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=32}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:18:16.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.144+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003540 con 0x55c136a11d60 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 10 ==== osd_map(32..35 src has 1..35) v4 ==== 4009+0+0 (secure 0 0 0) 0x7f50a8095830 con 0x55c136a4c5c0 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(32..35 src has 1..35) v4 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [32,35] > 31 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 32 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 32 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 33 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 33 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 34 2023-12-11T20:18:16.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:21.172 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 34 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:21.172 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:21.173 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 35 2023-12-11T20:18:21.173 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:21.173 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 35 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:21.173 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:21.173 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:21.173 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:16.176+0000 7f50a77fe640 20 client.4407.objecter 2 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:18:21.173 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:17.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:21.174 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:17.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:21.174 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:17.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:21.174 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:18.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:21.174 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:18.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:21.174 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:18.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:21.174 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:19.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:21.174 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:19.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:21.174 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:19.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:21.175 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:19.012+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:18:21.175 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:20.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:21.175 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:20.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:21.175 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:20.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:21.175 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:21.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:21.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:21.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.145+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:21.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.145+0000 7f50b85ed640 2 client.4407.objecter tid 2 on osd.6 is laggy 2023-12-11T20:18:21.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.145+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:21.176 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.145+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:18:21.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.145+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:21.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.145+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=36}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:18:21.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.145+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f50880037e0 con 0x55c136a11d60 2023-12-11T20:18:21.177 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.172+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 11 ==== osd_map(36..39 src has 1..39) v4 ==== 3284+0+0 (secure 0 0 0) 0x7f50a8099380 con 0x55c136a4c5c0 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.172+0000 7f50a77fe640 1 pool: Running main() from gmock_main.cc 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [==========] Running 6 tests from 1 test suite. 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [----------] Global test environment set-up. 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [----------] 6 tests from NeoRadosPools 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ OK ] NeoRadosPools.PoolList (6129 ms) 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4258 ms) 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3634 ms) 2023-12-11T20:18:22.257 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (7347 ms) 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1994 ms) 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2537 ms) 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [----------] 6 tests from NeoRadosPools (25899 ms total) 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [----------] Global test environment tear-down 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [==========] 6 tests from 1 test suite ran. (25899 ms total) 2023-12-11T20:18:22.258 INFO:tasks.workunit.client.0.smithi086.stdout: pool: [ PASSED ] 6 tests. 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: Running main() from gmock_main.cc 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [----------] Global test environment set-up. 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [----------] 1 test from neocls_handler_error 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (28304 ms) 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [----------] 1 test from neocls_handler_error (28304 ms total) 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [----------] Global test environment tear-down 2023-12-11T20:18:24.615 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [==========] 1 test from 1 test suite ran. (28304 ms total) 2023-12-11T20:18:24.616 INFO:tasks.workunit.client.0.smithi086.stdout: handler_error: [ PASSED ] 1 test. 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout:0 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(36..39 src has 1..39) v4 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.172+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [36,39] > 35 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.172+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 36 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 36 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 37 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:26.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 37 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 38 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 38 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 39 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 39 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:21.173+0000 7f50a77fe640 20 client.4407.objecter 2 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:22.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:22.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:22.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:26.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:23.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:23.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:23.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:24.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:24.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:24.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:25.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:25.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:25.012+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:26.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.012+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.012+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.013+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.146+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.146+0000 7f50b85ed640 2 client.4407.objecter tid 2 on osd.6 is laggy 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.146+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.146+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.146+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.146+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=40}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.146+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003a20 con 0x55c136a11d60 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 12 ==== osd_map(40..43 src has 1..43) v4 ==== 3609+0+0 (secure 0 0 0) 0x7f50a80995f0 con 0x55c136a4c5c0 2023-12-11T20:18:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(40..43 src has 1..43) v4 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [40,43] > 39 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 40 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 40 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 41 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 41 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 42 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 42 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 43 2023-12-11T20:18:26.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:26.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 43 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:26.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:26.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:26.735 INFO:tasks.workunit.client.0.smithi086.stdout: api_asi api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-12-11T20:18:26.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [----------] Global test environment set-up. 2023-12-11T20:18:26.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-12-11T20:18:26.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-12-11T20:18:26.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5225 ms) 2023-12-11T20:18:26.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-12-11T20:18:26.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (84 ms) 2023-12-11T20:18:26.736 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-12-11T20:18:26.737 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20055 ms) 2023-12-11T20:18:26.737 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-12-11T20:18:26.737 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: attempt 0 of 20 2023-12-11T20:18:26.737 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: failed to deregister: 2023-12-11T20:18:26.737 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: { 2023-12-11T20:18:26.737 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "epoch": 13, 2023-12-11T20:18:26.737 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "modified": "2023-12-11T20:18:22.565074+0000", 2023-12-11T20:18:26.737 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "services": { 2023-12-11T20:18:26.738 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "laundry": { 2023-12-11T20:18:26.738 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "daemons": { 2023-12-11T20:18:26.738 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "summary": "", 2023-12-11T20:18:26.738 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "close-test-pid92892": { 2023-12-11T20:18:26.738 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "start_epoch": 13, 2023-12-11T20:18:26.738 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "start_stamp": "2023-12-11T20:18:21.629678+0000", 2023-12-11T20:18:26.738 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "gid": 4825, 2023-12-11T20:18:26.739 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "addr": "172.21.15.86:0/2471819707", 2023-12-11T20:18:26.739 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "metadata": { 2023-12-11T20:18:26.739 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "arch": "x86_64", 2023-12-11T20:18:26.739 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "ceph_release": "reef", 2023-12-11T20:18:26.739 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-83-gc56ad3ea (c56ad3eaf0784383bf22c04e35d26e15c4baeafe) reef (dev)", 2023-12-11T20:18:26.739 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "ceph_version_short": "19.0.0-83-gc56ad3ea", 2023-12-11T20:18:26.739 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-11T20:18:26.740 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "distro": "centos", 2023-12-11T20:18:26.740 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-11T20:18:26.740 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "distro_version": "9", 2023-12-11T20:18:26.740 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "foo": "bar", 2023-12-11T20:18:26.741 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "hostname": "smithi086", 2023-12-11T20:18:26.741 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023", 2023-12-11T20:18:26.741 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "kernel_version": "5.14.0-390.el9.x86_64", 2023-12-11T20:18:26.741 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-11T20:18:26.741 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "mem_total_kb": "32488612", 2023-12-11T20:18:26.741 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "os": "Linux", 2023-12-11T20:18:26.742 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "this": "that" 2023-12-11T20:18:26.742 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: }, 2023-12-11T20:18:26.742 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "task_status": {} 2023-12-11T20:18:26.742 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: }, 2023-12-11T20:18:26.743 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "pid92847": { 2023-12-11T20:18:26.743 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "start_epoch": 12, 2023-12-11T20:18:26.743 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "start_stamp": "2023-12-11T20:18:08.919336+0000", 2023-12-11T20:18:26.744 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "gid": 4786, 2023-12-11T20:18:26.744 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "addr": "172.21.15.86:0/1614215636", 2023-12-11T20:18:26.744 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "metadata": { 2023-12-11T20:18:26.744 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "arch": "x86_64", 2023-12-11T20:18:26.744 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "ceph_release": "reef", 2023-12-11T20:18:26.744 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-83-gc56ad3ea (c56ad3eaf0784383bf22c04e35d26e15c4baeafe) reef (dev)", 2023-12-11T20:18:26.744 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "ceph_version_short": "19.0.0-83-gc56ad3ea", 2023-12-11T20:18:26.744 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-11T20:18:28.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_ser0 GiB avail 2023-12-11T20:18:28.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: pgs: 44.362% pgs unknown 2023-12-11T20:18:28.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 48.928% pgs not active 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 476 unknown 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 323 creating+peering 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 202 creating+activating 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 72 active+clean 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: io: 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: client: 0 B/s rd, 189 B/s wr, 0 op/s rd, 0 op/s wr 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5965 ms) 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ RUN ] LibRadosService.Status 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ OK ] LibRadosService.Status (20762 ms) 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [----------] 4 tests from LibRadosService (32770 ms total) 2023-12-11T20:18:28.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: 2023-12-11T20:18:28.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [----------] Global test environment tear-down 2023-12-11T20:18:28.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [==========] 4 tests from 1 test suite ran. (32770 ms total) 2023-12-11T20:18:28.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_service: [ PASSED ] 4 tests. 2023-12-11T20:18:29.371 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:18:29.310+0000 10a54640 -1 osd.5 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4470.0:2 29.6 29:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:29.455 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:29.350+0000 10253640 -1 osd.6 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:30.253 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:30.234+0000 10253640 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:30.625 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:30.558+0000 10253640 -1 osd.6 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:30.653 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:18:30.533+0000 10a54640 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4470.0:2 29.6 29:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:31.227 INFO:tasks.workunit.client.0.smithi086.stdout:o: 2023-12-11T20:18:26.161+0000 7f50a77fe640 20 client.4407.objecter 2 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:18:31.227 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:27.013+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:31.227 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:27.013+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:31.227 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:27.013+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:31.227 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:28.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:31.227 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:28.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:28.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:29.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:29.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:29.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:29.014+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:30.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:30.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:30.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.147+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.147+0000 7f50b85ed640 2 client.4407.objecter tid 2 on osd.6 is laggy 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.147+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:31.228 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.147+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.147+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.147+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=44}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.147+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003c10 con 0x55c136a11d60 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 13 ==== osd_map(44..47 src has 1..47) v4 ==== 1826+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(44..47 src has 1..47) v4 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [44,47] > 43 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 44 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 44 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:31.229 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:31.319 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:31.311+0000 10253640 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:31.559 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:31.551+0000 10253640 -1 osd.6 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:31.821 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:18:31.818+0000 10a54640 -1 osd.5 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4470.0:2 29.6 29:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incrvice_pp: "distro": "centos", 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "distro_version": "9", 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "foo": "bar", 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "hostname": "smithi086", 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023", 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "kernel_version": "5.14.0-390.el9.x86_64", 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-11T20:18:32.479 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "mem_total_kb": "32488612", 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "os": "Linux", 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "this": "that" 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: }, 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: "task_status": {} 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: } 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: } 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: } 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: } 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: } 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: attempt 1 of 20 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10949 ms) 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (36313 ms total) 2023-12-11T20:18:32.480 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: 2023-12-11T20:18:32.481 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [----------] Global test environment tear-down 2023-12-11T20:18:32.481 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (36313 ms total) 2023-12-11T20:18:32.481 INFO:tasks.workunit.client.0.smithi086.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-12-11T20:18:32.545 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:32.480+0000 10253640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:32.763 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:02:45.871 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:32.841 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:02:45.950 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:32.849 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:32.750+0000 10253640 -1 osd.6 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:32.861 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:02:45.969 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:32.904 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:45.973 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:32.907 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:45.975 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:32.910 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:02:46.018 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:33.047 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:02:46.156 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:33.123 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:46.191 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:33.179 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:02:46.287 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:33.257 INFO:tasks.workunit.client.0.smithi086.stdout:emental epoch 45 2023-12-11T20:18:33.257 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:33.257 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 45 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:33.257 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:33.257 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 46 2023-12-11T20:18:33.257 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 46 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 47 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 10 client.4407.objecter checking op 2 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 47 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:31.229+0000 7f50a77fe640 20 client.4407.objecter 2 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:32.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:32.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:32.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:33.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 2 ==== osd_op_reply(2 exist [writefull 0~5] v26'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(2 exist [writefull 0~5] v26'1 uv1 ondisk = 0) v8 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 2 ondisk uv 1 in 8.1 attempt 0 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 2 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 2 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 2 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-12-11T20:18:33.259 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.258+0000 7f50b9045780 10 librados: Objecter returned from writefull r=0 2023-12-11T20:18:33.627 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:33.624+0000 10253640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:33.754 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:18:33.743+0000 10a54640 -1 osd.5 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4470.0:2 29.6 29:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:33.943 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:33.859+0000 10253640 -1 osd.6 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:34.668 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:34.623+0000 10a54640 -1 osd.4 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:34.693 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:34.683+0000 10a54640 -1 osd.3 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:34.996 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:18:34.928+0000 10a54640 -1 osd.5 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4470.0:2 29.6 29:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:35.012 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136a112a0 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 47 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [read 0~256] tid 3 osd.6 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 3 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 15 client.4407.objecter _send_op 3 to 8.1 on osd.6 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:3 8.1 8:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47) v8 -- 0x55c136ac4690 con 0x55c136a11d60 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136acf700 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 47 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:35.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 10 client.4407.objecter _op_submit oid noexist '@8' '@8' [read 0~256] tid 4 osd.6 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 4 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 15 client.4407.objecter _send_op 4 to 8.1 on osd.6 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:4 8.1 8:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47) v8 -- 0x55c136ac49f0 con 0x55c136a11d60 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:33.259+0000 7f50b9045780 5 client.4407.objecter 2 in flight 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:34.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:34.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:34.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:35.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:35.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:35.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:35.120 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:18:35.114+0000 10253640 -1 osd.1 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 28.b 28:d4c8d119:test-rados-api-smithi086-92941-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-11T20:18:35.191 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:35.124+0000 10253640 -1 osd.6 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:35.313 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:35.209+0000 10253640 -1 osd.2 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:35.810 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:35.798+0000 10a54640 -1 osd.3 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:35.990 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:18:35.950+0000 10a54640 -1 osd.5 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4470.0:2 29.6 29:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:36.066 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:36.048+0000 10a54640 -1 osd.4 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:36.398 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:36.294+0000 10253640 -1 osd.6 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:36.663 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:18:36.636+0000 10253640 -1 osd.1 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 28.b 28:d4c8d119:test-rados-api-smithi086-92941-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-11T20:18:37.003 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:36.887+0000 10a54640 -1 osd.3 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:37.062 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:36.954+0000 10253640 -1 osd.2 48 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:37.219 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:18:37.209+0000 10a54640 -1 osd.5 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4470.0:2 29.6 29:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:37.252 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:37.251+0000 10a54640 -1 osd.4 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:37.586 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:37.579+0000 10253640 -1 osd.6 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:37.753 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:18:37.738+0000 10253640 -1 osd.1 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 28.b 28:d4c8d119:test-rados-api-smithi086-92941-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-11T20:18:38.110 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:37.996+0000 10a54640 -1 osd.3 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:38.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:35.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:38.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:36.014+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:38.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:36.014+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:38.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:36.014+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:38.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:36.148+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:38.143 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:37.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:37.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:37.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.144+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 3 ==== osd_map(48..48 src has 1..49) v4 ==== 636+0+0 (crc 0 0 0) 0x7f50b000dc80 con 0x55c136a11d60 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.144+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.144+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(48..48 src has 1..49) v4 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [48,48] > 47 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 48 2023-12-11T20:18:38.144 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 10 client.4407.objecter checking op 3 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 48 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 10 client.4407.objecter checking op 4 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 48 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 20 client.4407.objecter 3 8.d3a59201 osd.6 exist [read 0~256] 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50a77fe640 20 client.4407.objecter 4 8.a7bbaa1 osd.6 noexist [read 0~256] 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 3 ondisk uv 1 in 8.1 attempt 0 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 5 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 3 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 3 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:18:38.145 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 3 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 5 client.4407.objecter 1 in flight 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 4 ondisk uv 0 in 8.1 attempt 0 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 4 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 4 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 4 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (4886 ms) 2023-12-11T20:18:38.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136a112a0 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 48 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [read 0~256] tid 5 osd.6 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 5 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 15 client.4407.objecter _send_op 5 to 8.1 on osd.6 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:5 8.1 8:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55c136acf700 con 0x55c136a11d60 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:18:38.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136acfba0 2023-12-11T20:18:38.292 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:18:38.291+0000 10a54640 -1 osd.5 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4470.0:2 29.6 29:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:38.662 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:38.659+0000 10a54640 -1 osd.4 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:38.810 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:18:38.801+0000 10253640 -1 osd.1 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 28.b 28:d4c8d119:test-rados-api-smithi086-92941-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-11T20:18:38.851 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:38.640+0000 10253640 -1 osd.2 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:38.884 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:38.845+0000 10253640 -1 osd.6 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:39.453 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:39.408+0000 10a54640 -1 osd.3 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:39.653 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:39.647+0000 10253640 -1 osd.2 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:39.673 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:39.665+0000 10a54640 -1 osd.4 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:39.879 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:18:39.880+0000 10253640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 28.b 28:d4c8d119:test-rados-api-smithi086-92941-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-11T20:18:39.964 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:39.927+0000 10253640 -1 osd.6 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:40.603 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:40.510+0000 10a54640 -1 osd.3 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:40.714 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:40.658+0000 10a54640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:40.939 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:40.811+0000 10253640 -1 osd.2 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:41.161 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:41.052+0000 10253640 -1 osd.6 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:41.319 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:18:41.267+0000 10253640 -1 osd.1 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 28.b 28:d4c8d119:test-rados-api-smithi086-92941-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-11T20:18:41.648 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:41.648+0000 10a54640 -1 osd.3 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:41.803 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:41.790+0000 10a54640 -1 osd.4 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:42.015 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:41.932+0000 10253640 -1 osd.2 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:42.414 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:42.407+0000 10253640 -1 osd.6 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:42.824 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:42.770+0000 10a54640 -1 osd.3 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:42.851 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:42.759+0000 10a54640 -1 osd.7 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:42.923 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:42.890+0000 10a54640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:43.489 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:43.376+0000 10253640 -1 osd.2 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:43.696 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:18:43.613+0000 10253640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4476.0:2 19.1 19:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:43.861 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:43.807+0000 10a54640 -1 osd.3 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:44.194 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:44.179+0000 10a54640 -1 osd.4 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:44.297 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:44.206+0000 10a54640 -1 osd.7 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:44.335 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:18:44.307+0000 10253640 -1 osd.0 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:3 10.9 10:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:44.723 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:44.673+0000 10253640 -1 osd.2 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:45.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 48 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 10 client.4407.objecter _op_submit oid noexist '@8' '@8' [read 0~256] tid 6 osd.6 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 6 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 15 client.4407.objecter _send_op 6 to 8.1 on osd.6 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:6 8.1 8:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55c136ad0040 con 0x55c136a11d60 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:38.145+0000 7f50b9045780 5 client.4407.objecter 2 in flight 2023-12-11T20:18:45.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:39.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:39.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:39.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:39.015+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:40.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:40.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:40.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:41.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:41.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:41.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:41.149+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:42.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:42.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:42.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:45.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:43.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:45.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:43.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:45.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:43.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:45.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:44.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:45.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:44.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:45.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:44.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:45.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:45.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:45.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:45.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:45.267 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:45.240+0000 10a54640 -1 osd.3 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:45.537 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:45.507+0000 10a54640 -1 osd.7 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:45.581 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:45.516+0000 10a54640 -1 osd.4 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:45.015+0000 7 api_misc: [==========] Running 12 tests from 4 test suites. 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] Global test environment set-up. 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-12-11T20:18:46.217 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: 2023-12-11T20:17:56.127+0000 7f2819246780 -1 failed for service _ceph-mon._tcp 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: 2023-12-11T20:17:56.127+0000 7f2819246780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (143 ms) 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2016 ms) 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2159 ms total) 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabcfeb340 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc2fcca0 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc672e00 2023-12-11T20:18:46.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc6737c0 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc674190 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc674b90 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc675590 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc675f90 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc676990 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc676fe0 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc6779c0 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc676790 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc6788b0 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc677710 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc679720 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc678620 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc67a240 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc67ac50 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc679540 2023-12-11T20:18:46.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc67bb40 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc67aa50 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc67ca70 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc67b8a0 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc483e30 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc4848c0 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc67c7f0 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc4857f0 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc484640 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc486680 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc485550 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc487510 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc486410 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc488010 2023-12-11T20:18:46.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc488ad0 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc487310 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc489960 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc4887e0 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabcfb12e0 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabcfb13f0 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabcfb1540 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabcfb16d0 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc4975e0 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc6775f0 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 0x55cabc488500 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: started 44 aios 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabcfeb340 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc2fcca0 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc672e00 2023-12-11T20:18:46.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc6737c0 2023-12-11T20:18:46.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc674190 2023-12-11T20:18:46.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc674b90 2023-12-11T20:18:46.236 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:46.166+0000 10253640 -1 osd.2 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:46.591 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:46.529+0000 10a54640 -1 osd.3 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:46.825 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:46.747+0000 10a54640 -1 osd.7 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:46.967 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:46.799+0000 10a54640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:47.395 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:47.383+0000 10253640 -1 osd.2 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:47.644 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:18:47.515+0000 10253640 -1 osd.0 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:3 10.9 10:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:47.678 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:47.588+0000 10a54640 -1 osd.3 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:47.852 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:47.846+0000 10a54640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:48.221 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:48.182+0000 10a54640 -1 osd.7 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:48.592 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:48.536+0000 10253640 -1 osd.2 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:48.763 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:18:48.675+0000 10253640 -1 osd.0 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:3 10.9 10:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:48.797 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:48.779+0000 10a54640 -1 osd.3 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:49.057 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:49.014+0000 10a54640 -1 osd.4 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16.81d714e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:49.378 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:49.298+0000 10a54640 -1 osd.7 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:49.666 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:18:49.635+0000 10253640 -1 osd.0 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:3 10.9 10:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:46.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:46.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:46.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:46.150+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:47.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:47.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:47.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:49.699 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:48.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:48.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:48.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.016+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 6 ==== osd_map(49..60 src has 1..60) v4 ==== 7935+0+0 (crc 0 0 0) 0x7f50b000dc80 con 0x55c136a11d60 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(49..60 src has 1..60) v4 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [49,60] > 48 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 49 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 49 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.700 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 49 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 50 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 50 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 50 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 51 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 51 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.701 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 51 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 52 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 52 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 52 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 53 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 53 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 53 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.702 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 54 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 54 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 54 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 55 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 55 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 55 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.703 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 56 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 56 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 56 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 57 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 57 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.704 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 57 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 58 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 58 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 58 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 59 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 59 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.705 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 59 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 60 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 5 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 60 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter checking op 6 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 60 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 monclient: _renew_subs 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=61}) v3 -- 0x7f509807a3c0 con 0x55c136a4c5c0 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:49.706 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter 5 8.d3a59201 osd.6 exist [read 0~256] 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50a77fe640 20 client.4407.objecter 6 8.a7bbaa1 osd.6 noexist [read 0~256] 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 7 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 5 ondisk uv 1 in 8.1 attempt 0 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 5 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 5 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 5 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 5 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 5 client.4407.objecter 1 in flight 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 8 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-11T20:18:49.707 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:18:49.708 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 6 ondisk uv 0 in 8.1 attempt 0 2023-12-11T20:18:49.708 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-11T20:18:49.708 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:18:49.708 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 6 2023-12-11T20:18:49.708 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 6 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 6 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (11556 ms) 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136acfba0 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 60 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [read 0~256] tid 7 osd.6 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 7 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.699+0000 7f50b9045780 15 client.4407.objecter _send_op 7 to 8.1 on osd.6 2023-12-11T20:18:49.709 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:7 8.1 8:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) v8 -- 0x55c136ad03a0 con 0x55c136a11d60 2023-12-11T20:18:49.710 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:49.710 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:18:49.710 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136a112a0 2023-12-11T20:18:49.710 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 60 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.710 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.710 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:18:49.710 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:49.719 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:49.683+0000 10253640 -1 osd.2 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:49.751 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 10 client.4407.objecter _op_submit oid noexist '@8' '@8' [read 0~256] tid 8 osd.6 2023-12-11T20:18:49.751 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 8 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 15 client.4407.objecter _send_op 8 to 8.1 on osd.6 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:8 8.1 8:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) v8 -- 0x55c136b80990 con 0x55c136a11d60 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.700+0000 7f50b9045780 5 client.4407.objecter 2 in flight 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 14 ==== osd_map(61..62 src has 1..62) v4 ==== 1608+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(61..62 src has 1..62) v4 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [61,62] > 60 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 61 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 10 client.4407.objecter checking op 7 2023-12-11T20:18:49.752 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 61 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 10 client.4407.objecter checking op 8 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 61 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 62 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 10 client.4407.objecter checking op 7 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 62 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 10 client.4407.objecter checking op 8 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 62 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:49.753 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 10 monclient: _renew_subs 2023-12-11T20:18:49.754 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:49.754 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=63}) v3 -- 0x7f509807a3c0 con 0x55c136a4c5c0 2023-12-11T20:18:49.808 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:49.801+0000 10a54640 -1 osd.3 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:50.478 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:50.429+0000 10a54640 -1 osd.4 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:50.636 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:50.636+0000 10a54640 -1 osd.7 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:50.864 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:50.834+0000 10253640 -1 osd.2 60 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:50.898 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:50.898 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter 7 8.d3a59201 osd.6 exist [read 0~256] 2023-12-11T20:18:50.898 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:49.752+0000 7f50a77fe640 20 client.4407.objecter 8 8.a7bbaa1 osd.6 noexist [read 0~256] 2023-12-11T20:18:50.898 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:50.898 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:50.898 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 9 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 7 ondisk uv 1 in 8.1 attempt 0 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 5 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 7 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 7 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1199 ms) 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 7 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 5 client.4407.objecter 1 in flight 2023-12-11T20:18:50.899 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 10 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 8 ondisk uv 0 in 8.1 attempt 0 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 8 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 8 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 8 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.898+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:18:50.900 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-11T20:18:50.956 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:50.878+0000 10a54640 -1 osd.3 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:51.011 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:18:51.000+0000 10253640 -1 osd.0 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:3 10.9 10:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:51.194 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 20 librados: queue_aio_write 0x55c1369c1480 completion 0x55c13695cca0 write_seq 1 2023-12-11T20:18:51.194 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136a112a0 2023-12-11T20:18:51.194 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 62 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [write 0~5 in=5b] tid 9 osd.6 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 9 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 15 client.4407.objecter _send_op 9 to 8.1 on osd.6 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:9 8.1 8:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62) v8 -- 0x55c136acfb80 con 0x55c136a11d60 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:50.899+0000 7f50b9045780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-11T20:18:51.195 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.150+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 15 ==== osd_map(63..63 src has 1..63) v4 ==== 2368+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(63..63 src has 1..63) v4 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [63,63] > 62 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 63 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 63 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 10 monclient: _renew_subs 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:51.196 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=64}) v3 -- 0x7f509807a3c0 con 0x55c136a4c5c0 2023-12-11T20:18:51.692 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:51.520+0000 10a54640 -1 osd.4 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:51.804 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:51.657+0000 10a54640 -1 osd.7 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:52.022 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:18:52.016+0000 10253640 -1 osd.0 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:3 10.9 10:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:52.225 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:52.215+0000 10a54640 -1 osd.3 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:52.693 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:52.683+0000 10a54640 -1 osd.4 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:53.110 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:52.989+0000 10a54640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:53.397 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:53.396+0000 10a54640 -1 osd.3 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:51.195+0000 7f50a77fe640 20 client.4407.objecter 9 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 16 ==== osd_map(64..64 src has 1..64) v4 ==== 652+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(64..64 src has 1..64) v4 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [64,64] > 63 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 64 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:18:53.679 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 64 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 10 monclient: _renew_subs 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=65}) v3 -- 0x7f509807a3c0 con 0x55c136a4c5c0 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:52.333+0000 7f50a77fe640 20 client.4407.objecter 9 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 17 ==== osd_map(65..65 src has 1..65) v4 ==== 1693+0+0 (secure 0 0 0) 0x7f50a80b4020 con 0x55c136a4c5c0 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(65..65 src has 1..65) v4 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [65,65] > 64 2023-12-11T20:18:53.680 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 65 2023-12-11T20:18:53.681 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:18:53.681 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 65 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:53.681 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:53.681 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:53.835 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77f api_pool: Running main() from gmock_main.cc 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [----------] Global test environment set-up. 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolList (9896 ms) 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4482 ms) 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5760 ms) 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-12-11T20:18:53.836 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3581 ms) 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3628 ms) 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6175 ms) 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (7398 ms) 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6260 ms) 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10654 ms) 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [----------] 9 tests from LibRadosPools (57835 ms total) 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [----------] Global test environment tear-down 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (57835 ms total) 2023-12-11T20:18:53.837 INFO:tasks.workunit.client.0.smithi086.stdout: api_pool: [ PASSED ] 9 tests. 2023-12-11T20:18:53.899 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:53.852+0000 10a54640 -1 osd.4 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:54.538 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:54.510+0000 10a54640 -1 osd.3 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:54.545 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:54.418+0000 10a54640 -1 osd.7 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:54.798 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:54.729+0000 10253640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:55.249 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:55.225+0000 10a54640 -1 osd.4 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:55.595 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:18:55.568+0000 10a54640 -1 osd.3 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:55.758 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:55.624+0000 10a54640 -1 osd.7 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout:e640 10 monclient: _renew_subs 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=66}) v3 -- 0x7f509807a3c0 con 0x55c136a4c5c0 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:53.679+0000 7f50a77fe640 20 client.4407.objecter 9 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 18 ==== osd_map(66..66 src has 1..66) v4 ==== 664+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(66..66 src has 1..66) v4 2023-12-11T20:18:55.874 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [66,66] > 65 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 66 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 66 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 10 monclient: _renew_subs 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=67}) v3 -- 0x7f509807a3c0 con 0x55c136a4c5c0 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:54.847+0000 7f50a77fe640 20 client.4407.objecter 9 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 19 ==== osd_map(67..67 src has 1..67) v4 ==== 312+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:18:55.875 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(67..67 src has 1..67) v4 2023-12-11T20:18:55.876 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [67,67] > 66 2023-12-11T20:18:55.876 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 67 2023-12-11T20:18:55.876 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:18:56.140 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:56.140+0000 10253640 -1 osd.2 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:56.314 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:56.310+0000 10a54640 -1 osd.4 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:57.189 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:57.125+0000 10a54640 -1 osd.7 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:57.424 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:57.336+0000 10253640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:57.514 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:57.513+0000 10a54640 -1 osd.4 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:58.306 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:58.291+0000 10a54640 -1 osd.7 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:58.523 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:58.511+0000 10253640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:18:58.883 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:11.991 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:58.888 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:11.997 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:58.948 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:18:58.901+0000 10a54640 -1 osd.4 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:18:58.956 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:12.024 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:58.958 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:12.027 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:58.972 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:12.080 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:58.977 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:12.086 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:59.112 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:12.181 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:59.116 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:12.225 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:59.169 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:12.278 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:18:59.453 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:18:59.432+0000 10a54640 -1 osd.7 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:18:59.587 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:18:59.582+0000 10253640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:19:00.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 67 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:00.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:00.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:19:00.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 10 monclient: _renew_subs 2023-12-11T20:19:00.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:19:00.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=68}) v3 -- 0x7f509807a3c0 con 0x55c136a4c5c0 2023-12-11T20:19:00.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:55.874+0000 7f50a77fe640 20 client.4407.objecter 9 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:56.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:56.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:56.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:56.151+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 20 ==== osd_map(68..68 src has 1..68) v4 ==== 1047+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(68..68 src has 1..68) v4 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [68,68] > 67 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 68 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:19:00.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 68 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:57.040+0000 7f50a77fe640 20 client.4407.objecter 9 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:58.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:58.015+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:58.015+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:59.015+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:59.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:59.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:18:59.016+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:00.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:00.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:00.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:00.822 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:00.798+0000 10a54640 -1 osd.7 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:00.881 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:19:00.879+0000 10253640 -1 osd.2 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:00.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.152+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.152+0000 7f50b85ed640 2 client.4407.objecter tid 9 on osd.6 is laggy 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.152+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.152+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.152+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.152+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=69}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:19:01.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.152+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003c10 con 0x55c136a11d60 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 21 ==== osd_map(69..72 src has 1..72) v4 ==== 2505+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(69..72 src has 1..72) v4 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [69,72] > 68 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 69 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 69 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 70 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 70 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 71 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 71 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:01.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:01.224 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 72 2023-12-11T20:19:01.224 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 10 client.4407.objecter checking op 9 2023-12-11T20:19:01.224 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 72 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:01.342 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:19:01.342+0000 10a54640 -1 osd.4 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:01.956 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:01.917+0000 10a54640 -1 osd.3 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:01.995 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:19:01.988+0000 10253640 -1 osd.2 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:19:02.292 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:02.283+0000 10a54640 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:02.438 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:19:02.375+0000 10a54640 -1 osd.4 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:02.451 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:02.295+0000 10253640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:02.916 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:02.905+0000 10a54640 -1 osd.3 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:03.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:03.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:03.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:01.222+0000 7f50a77fe640 20 client.4407.objecter 9 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:19:03.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:02.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:03.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:02.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:03.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:02.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:03.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:03.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 11 ==== osd_op_reply(9 exist [write 0~5] v62'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(9 exist [write 0~5] v62'2 uv2 ondisk = 0) v8 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 9 ondisk uv 2 in 8.1 attempt 0 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 9 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 9 2023-12-11T20:19:03.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 9 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b6b8a640 20 librados: complete_aio_write 0x55c13695cca0 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (12233 ms) 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 20 librados: queue_aio_write 0x55c1369c1480 completion 0x55c136969660 write_seq 2 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136a112a0 2023-12-11T20:19:03.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 72 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:03.134 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:03.134 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:03.134 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:03.134 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [write 0~5 in=5b] tid 10 osd.6 2023-12-11T20:19:03.134 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:03.134 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 10 2023-12-11T20:19:03.142 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:19:03.142+0000 10253640 -1 osd.2 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:19:03.300 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:03.295+0000 10a54640 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:03.427 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:03.363+0000 10253640 -1 osd.0 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:03.593 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:19:03.586+0000 10a54640 -1 osd.4 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:03.952 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:03.952+0000 10a54640 -1 osd.3 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:04.617 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:19:04.605+0000 10253640 -1 osd.2 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:19:04.677 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:19:04.673+0000 10a54640 -1 osd.4 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:04.838 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:04.740+0000 10253640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:04.968 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:04.920+0000 10a54640 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:05.220 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:05.218+0000 10a54640 -1 osd.3 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:05.819 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:19:05.813+0000 10a54640 -1 osd.4 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4627.0:2 16.4 16:2728eb81:test-rados-api-smithi086-92766-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:05.994 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:05.960+0000 10a54640 -1 osd.7 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:06.030 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:05.896+0000 10253640 -1 osd.0 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:06.042 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:19:05.940+0000 10253640 -1 osd.2 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4461.0:2 18.6 18:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-11T20:19:06.441 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:06.404+0000 10a54640 -1 osd.3 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:07.023 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:06.969+0000 10a54640 -1 osd.7 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:07.125 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:07.115+0000 10253640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:07.451 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:07.430+0000 10a54640 -1 osd.3 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:08.186 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:08.177+0000 10a54640 -1 osd.7 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:08.445 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:08.409+0000 10a54640 -1 osd.3 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 15 client.4407.objecter _send_op 10 to 8.1 on osd.6 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:10 8.1 8:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) v8 -- 0x55c136b80320 con 0x55c136a11d60 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:03.131+0000 7f50b9045780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:09.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 12 ==== osd_map(73..73 src has 1..73) v4 ==== 304+0+0 (crc 0 0 0) 0x7f50b000dc80 con 0x55c136a11d60 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(73..73 src has 1..73) v4 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [73,73] > 72 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 73 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 10 client.4407.objecter checking op 10 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 73 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:04.981+0000 7f50a77fe640 20 client.4407.objecter 10 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:05.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:05.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:05.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:06.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:06.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:06.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:09.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:06.153+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:09.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:07.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:09.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:07.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:09.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:07.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:09.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:08.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:09.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:08.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:09.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:08.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:09.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:09.016+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:09.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:09.016+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:09.569 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:09.401+0000 10a54640 -1 osd.7 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:09.572 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:09.459+0000 10a54640 -1 osd.3 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:10.464 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:10.458+0000 10a54640 -1 osd.3 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:10.536 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:10.501+0000 10a54640 -1 osd.7 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:11.494 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:11.484+0000 10a54640 -1 osd.3 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:11.674 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:11.514+0000 10a54640 -1 osd.7 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:09.016+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:09.016+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:10.017+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:10.017+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:10.017+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.017+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.017+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.017+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.154+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 13 ==== osd_op_reply(10 exist [write 0~5] v73'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:11.847 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(10 exist [write 0~5] v73'3 uv3 ondisk = 0) v8 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 10 ondisk uv 3 in 8.1 attempt 0 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 10 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 10 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 10 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b6b8a640 20 librados: complete_aio_write 0x55c136969660 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (8716 ms) 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 20 librados: queue_aio_write 0x55c1369c1480 completion 0x55c136b80320 write_seq 3 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136a112a0 2023-12-11T20:19:11.848 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 73 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:11.849 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:11.849 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:11.849 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:11.849 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [write 0~5 in=5b] tid 11 osd.6 2023-12-11T20:19:11.849 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:11.849 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 11 2023-12-11T20:19:12.853 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:12.743+0000 10a54640 -1 osd.3 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:13.807 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:13.808+0000 10a54640 -1 osd.3 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:15.174 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:15.170+0000 10a54640 -1 osd.3 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:15.365 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:15.361+0000 10a54640 -1 osd.7 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:16.327 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:16.194+0000 10a54640 -1 osd.3 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:16.843 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:16.836+0000 10a54640 -1 osd.7 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:17.209 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 15 client.4407.objecter _send_op 11 to 8.1 on osd.6 2023-12-11T20:19:17.209 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:11 8.1 8:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73) v8 -- 0x55c136b840f0 con 0x55c136a11d60 2023-12-11T20:19:17.209 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:17.209 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:19:17.209 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:11.847+0000 7f50b9045780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-11T20:19:17.209 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:12.017+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:12.017+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:12.017+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:13.017+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:13.017+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:13.017+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:14.018+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:14.018+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:14.018+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:15.018+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:15.018+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:15.018+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:16.018+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:17.210 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:16.018+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:16.018+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:16.155+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.018+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.018+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.018+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 14 ==== osd_map(74..82 src has 1..83) v4 ==== 7871+0+0 (crc 0 0 0) 0x7f50b000dc80 con 0x55c136a11d60 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(74..82 src has 1..83) v4 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [74,82] > 73 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 74 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 74 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.211 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 75 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 75 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 76 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 76 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 77 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 77 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 78 2023-12-11T20:19:17.212 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 78 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 79 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 79 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 80 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 80 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 81 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 81 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.213 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:17.214 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 82 2023-12-11T20:19:17.214 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:17.214 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 82 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:17.272 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:17.261+0000 10a54640 -1 osd.3 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:17.819 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:17.814+0000 10a54640 -1 osd.7 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:18.366 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:18.293+0000 10a54640 -1 osd.3 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:18.996 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:18.882+0000 10a54640 -1 osd.7 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:19.081 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:32.190 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:19.166 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:32.274 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:19.182 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:32.251 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:19.189 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:32.258 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:19.222 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:32.330 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:19.224 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:32.333 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:19.285 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:19.287+0000 10a54640 -1 osd.3 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:19.313 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:32.381 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:19.339 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:32.448 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:19.343 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:32.452 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:20.038 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:19.926+0000 10a54640 -1 osd.7 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:21.009 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:21.008+0000 10a54640 -1 osd.7 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:22.418 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:22.397+0000 10a54640 -1 osd.7 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:23.487 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:19:23.482+0000 10a54640 -1 osd.7 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4508.0:2 49.c 49:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2023-12-11T20:19:24.098 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:24.093+0000 10253640 -1 osd.0 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:24.307 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:24.270+0000 10a54640 -1 osd.3 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:25.161 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:25.162+0000 10253640 -1 osd.0 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:25.232 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:25.234+0000 10a54640 -1 osd.3 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:17.211+0000 7f50a77fe640 20 client.4407.objecter 11 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:18.018+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:18.018+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:18.018+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:19.018+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:19.018+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.155 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:19.018+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:19.018+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:20.018+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:20.018+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:20.018+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:21.018+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:21.018+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:21.018+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:21.156+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:22.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:22.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:22.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:23.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:23.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:23.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.156 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:24.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:24.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:24.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:25.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:25.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:25.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.157+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.157+0000 7f50b85ed640 2 client.4407.objecter tid 11 on osd.6 is laggy 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.157+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.157+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:19:26.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.157+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:19:26.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.157+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=83}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:19:26.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.157+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003c10 con 0x55c136a11d60 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 22 ==== osd_map(83..94 src has 1..94) v4 ==== 8428+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(83..94 src has 1..94) v4 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [83,94] > 82 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 83 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 83 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 84 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 84 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 85 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:26.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 85 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 86 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 86 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 87 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 87 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 88 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:26.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 88 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:26.339 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:26.339+0000 10a54640 -1 osd.3 93 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:27.434 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:27.428+0000 10a54640 -1 osd.3 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:28.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:28.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 89 2023-12-11T20:19:28.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 89 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.162+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 90 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 90 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 91 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 91 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 92 2023-12-11T20:19:28.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 92 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 93 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 93 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 94 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 94 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:26.163+0000 7f50a77fe640 20 client.4407.objecter 11 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:27.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:27.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:28.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:27.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:28.510 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:28.509+0000 10a54640 -1 osd.3 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:29.676 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:29.661+0000 10a54640 -1 osd.3 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:30.683 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:30.678+0000 10a54640 -1 osd.3 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:28.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:28.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:28.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:29.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:29.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:29.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:29.019+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:30.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:31.178 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:30.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:30.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.158+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.158+0000 7f50b85ed640 2 client.4407.objecter tid 11 on osd.6 is laggy 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.158+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.158+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.158+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.158+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=95}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.158+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003c10 con 0x55c136a11d60 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 23 ==== osd_map(95..99 src has 1..99) v4 ==== 2734+0+0 (secure 0 0 0) 0x7f50a80b4020 con 0x55c136a4c5c0 2023-12-11T20:19:31.179 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(95..99 src has 1..99) v4 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [95,99] > 94 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 95 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 95 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 96 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 96 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:31.180 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:31.727 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:31.726+0000 10a54640 -1 osd.3 98 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:32.991 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:32.873+0000 10a54640 -1 osd.3 100 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:34.970 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 97 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 97 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 98 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 98 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 99 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 10 client.4407.objecter checking op 11 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 99 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:34.971 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:31.179+0000 7f50a77fe640 20 client.4407.objecter 11 8.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:32.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:32.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:32.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:33.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:33.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:33.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 15 ==== osd_op_reply(11 exist [write 0~5] v82'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(11 exist [write 0~5] v82'4 uv4 ondisk = 0) v8 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 11 ondisk uv 4 in 8.1 attempt 0 2023-12-11T20:19:34.972 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:19:34.973 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 11 2023-12-11T20:19:34.973 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 11 2023-12-11T20:19:34.973 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:34.973 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 11 2023-12-11T20:19:35.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:19:35.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b6b8a640 20 librados: complete_aio_write 0x55c136b80320 2023-12-11T20:19:35.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (23124 ms) 2023-12-11T20:19:35.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-12-11T20:19:35.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136a112a0 2023-12-11T20:19:35.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 99 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:35.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:35.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:35.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [read 0~0] tid 12 osd.6 2023-12-11T20:19:35.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:35.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 12 2023-12-11T20:19:35.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 15 client.4407.objecter _send_op 12 to 8.1 on osd.6 2023-12-11T20:19:35.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:12 8.1 8:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c136b83a80 con 0x55c136a11d60 2023-12-11T20:19:35.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:19:35.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136b83de0 2023-12-11T20:19:35.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 99 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:19:35.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:35.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 10 client.4407.objecter _op_submit oid noexist '@8' '@8' [read 0~0] tid 13 osd.6 2023-12-11T20:19:35.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 13 2023-12-11T20:19:35.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 15 client.4407.objecter _send_op 13 to 8.1 on osd.6 2023-12-11T20:19:35.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:13 8.1 8:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c136b80cf0 con 0x55c136a11d60 2023-12-11T20:19:35.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:19:35.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:34.972+0000 7f50b9045780 5 client.4407.objecter 2 in flight 2023-12-11T20:19:35.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:35.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:35.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:35.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 16 ==== osd_map(100..101 src has 1..102) v4 ==== 588+0+0 (crc 0 0 0) 0x7f50b000dc80 con 0x55c136a11d60 2023-12-11T20:19:35.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 17 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:35.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(100..101 src has 1..102) v4 2023-12-11T20:19:35.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-11T20:19:35.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:35.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [100,101] > 99 2023-12-11T20:19:35.089 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 100 2023-12-11T20:19:35.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 10 client.4407.objecter checking op 12 2023-12-11T20:19:35.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 100 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:35.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 10 client.4407.objecter checking op 13 2023-12-11T20:19:35.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 100 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.090 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 101 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 10 client.4407.objecter checking op 12 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 101 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 10 client.4407.objecter checking op 13 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 101 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter 12 8.d3a59201 osd.6 exist [read 0~0] 2023-12-11T20:19:35.091 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50a77fe640 20 client.4407.objecter 13 8.a7bbaa1 osd.6 noexist [read 0~0] 2023-12-11T20:19:35.092 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.089+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 12 ondisk uv 4 in 8.1 attempt 0 2023-12-11T20:19:35.092 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.090+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 5 2023-12-11T20:19:35.092 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.090+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 12 2023-12-11T20:19:35.107 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.090+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 12 2023-12-11T20:19:35.107 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.090+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:19:35.107 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.090+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 12 2023-12-11T20:19:35.107 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.090+0000 7f50b6b8a640 5 client.4407.objecter 1 in flight 2023-12-11T20:19:35.107 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 18 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:35.107 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-11T20:19:35.108 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:35.108 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 13 ondisk uv 0 in 8.1 attempt 0 2023-12-11T20:19:35.108 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-11T20:19:35.108 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:19:35.108 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 13 2023-12-11T20:19:35.108 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 13 2023-12-11T20:19:35.108 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.108 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 13 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (136 ms) 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136b83a80 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 101 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [read 0~0] tid 14 osd.6 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 14 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 15 client.4407.objecter _send_op 14 to 8.1 on osd.6 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:14 8.1 8:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101) v8 -- 0x55c136a113b0 con 0x55c136a11d60 2023-12-11T20:19:35.109 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.110 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:19:35.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136b80cf0 2023-12-11T20:19:35.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 101 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:19:35.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:35.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.218 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 10 client.4407.objecter _op_submit oid noexist '@8' '@8' [read 0~0] tid 15 osd.6 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 15 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 15 client.4407.objecter _send_op 15 to 8.1 on osd.6 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:15 8.1 8:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101) v8 -- 0x55c136b84030 con 0x55c136a11d60 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.109+0000 7f50b9045780 5 client.4407.objecter 2 in flight 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 19 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 14 ondisk uv 4 in 8.1 attempt 0 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 5 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 14 2023-12-11T20:19:35.219 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 14 2023-12-11T20:19:35.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:19:35.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 14 2023-12-11T20:19:35.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 5 client.4407.objecter 1 in flight 2023-12-11T20:19:35.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 20 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:35.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-11T20:19:35.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:35.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 15 ondisk uv 0 in 8.1 attempt 0 2023-12-11T20:19:35.220 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-11T20:19:35.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:19:35.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 15 2023-12-11T20:19:35.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 15 2023-12-11T20:19:35.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 15 2023-12-11T20:19:35.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:19:35.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (111 ms) 2023-12-11T20:19:35.221 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136b80cf0 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 101 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [read 0~0] tid 16 osd.6 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.219+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 16 2023-12-11T20:19:35.222 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 15 client.4407.objecter _send_op 16 to 8.1 on osd.6 2023-12-11T20:19:35.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:16 8.1 8:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101) v8 -- 0x55c136b840f0 con 0x55c136a11d60 2023-12-11T20:19:35.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:19:35.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136b812c0 2023-12-11T20:19:35.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 101 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:19:35.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:35.223 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.224 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 10 client.4407.objecter _op_submit oid noexist '@8' '@8' [read 0~0] tid 17 osd.6 2023-12-11T20:19:35.224 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:35.224 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 17 2023-12-11T20:19:35.224 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 15 client.4407.objecter _send_op 17 to 8.1 on osd.6 2023-12-11T20:19:35.974 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:17 8.1 8:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101) v8 -- 0x55c136b81760 con 0x55c136a11d60 2023-12-11T20:19:35.974 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:19:35.974 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.220+0000 7f50b9045780 5 client.4407.objecter 2 in flight 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 21 ==== osd_map(102..102 src has 1..102) v4 ==== 296+0+0 (crc 0 0 0) 0x7f50b0029560 con 0x55c136a11d60 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 22 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(102..102 src has 1..102) v4 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [102,102] > 101 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 102 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 10 client.4407.objecter checking op 16 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 102 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 10 client.4407.objecter checking op 17 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 102 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 20 client.4407.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2023-12-11T20:19:35.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 20 client.4407.objecter 16 8.d3a59201 osd.6 exist [read 0~0] 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50a77fe640 20 client.4407.objecter 17 8.a7bbaa1 osd.6 noexist [read 0~0] 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 16 ondisk uv 4 in 8.1 attempt 0 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 5 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 16 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 16 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 5 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 16 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 5 client.4407.objecter 1 in flight 2023-12-11T20:19:35.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 23 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8 cls: Running main() from gmock_main.cc 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [==========] Running 2 tests from 1 test suite. 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [----------] Global test environment set-up. 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [----------] 2 tests from NeoRadosCls 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [ RUN ] NeoRadosCls.DNE 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [ OK ] NeoRadosCls.DNE (49664 ms) 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (50309 ms) 2023-12-11T20:19:36.258 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [----------] 2 tests from NeoRadosCls (99973 ms total) 2023-12-11T20:19:36.259 INFO:tasks.workunit.client.0.smithi086.stdout: cls: 2023-12-11T20:19:36.259 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [----------] Global test environment tear-down 2023-12-11T20:19:36.259 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [==========] 2 tests from 1 test suite ran. (99973 ms total) 2023-12-11T20:19:36.259 INFO:tasks.workunit.client.0.smithi086.stdout: cls: [ PASSED ] 2 tests. 2023-12-11T20:19:36.348 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:36.270+0000 10a54640 -1 osd.3 103 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:37.018 INFO:tasks.workunit.client.0.smithi086.stdout:a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-11T20:19:37.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:37.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 17 ondisk uv 0 in 8.1 attempt 0 2023-12-11T20:19:37.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-11T20:19:37.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 17 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 17 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 17 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (757 ms) 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 20 librados: queue_aio_write 0x55c1369c1480 completion 0x55c136b844e0 write_seq 4 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136b80cf0 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 102 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [writefull 0~5 in=5b] tid 18 osd.6 2023-12-11T20:19:37.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 18 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 15 client.4407.objecter _send_op 18 to 8.1 on osd.6 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:18 8.1 8:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) v8 -- 0x55c136b83a80 con 0x55c136a11d60 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:35.975+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:36.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:36.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:36.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:36.158+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:37.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:37.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:37.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:37.577 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:37.529+0000 10a54640 -1 osd.3 104 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:38.723 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:38.614+0000 10a54640 -1 osd.3 105 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:39.038 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:52.146 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.042 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:52.151 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.110 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:52.218 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.111 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:52.220 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.118 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:52.186 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.122 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:52.191 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.148 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:03:52.256 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.206 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:52.275 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.212 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:03:52.320 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:19:39.792 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:39.705+0000 10a54640 -1 osd.3 106 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:40.807 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:40.806+0000 10a54640 -1 osd.3 107 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:42.052 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:41.880+0000 10a54640 -1 osd.3 108 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:42.902 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:42.885+0000 10a54640 -1 osd.3 108 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:43.284 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:43.250+0000 10253640 -1 osd.0 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:43.926 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:43.926+0000 10a54640 -1 osd.3 109 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:44.411 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:19:44.411+0000 10253640 -1 osd.0 109 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 56.0 56.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-11T20:19:45.256 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:45.176+0000 10a54640 -1 osd.3 111 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:46.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:37.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:38.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:38.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:38.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:39.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:39.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:39.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:39.019+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:40.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:40.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:40.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:41.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:41.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:41.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:41.159+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:42.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:42.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:42.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:43.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:43.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:43.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:44.019+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:44.019+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:44.019+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:45.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:45.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:45.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.160+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.160+0000 7f50b85ed640 2 client.4407.objecter tid 18 on osd.6 is laggy 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.160+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.160+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.160+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:19:46.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.160+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=103}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.160+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003c10 con 0x55c136a11d60 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 24 ==== osd_map(103..112 src has 1..112) v4 ==== 4413+0+0 (secure 0 0 0) 0x7f50a80a0f90 con 0x55c136a4c5c0 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(103..112 src has 1..112) v4 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [103,112] > 102 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 103 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 103 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 104 2023-12-11T20:19:46.162 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 104 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 105 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 105 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 106 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 106 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 107 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 107 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:46.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 108 2023-12-11T20:19:46.164 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:46.164 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 108 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:46.164 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:46.208 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:46.208+0000 10a54640 -1 osd.3 111 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:47.405 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:47.242+0000 10a54640 -1 osd.3 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:48.360 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:48.354+0000 10a54640 -1 osd.3 114 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:48.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 109 2023-12-11T20:19:48.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:48.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 109 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:48.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:48.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 110 2023-12-11T20:19:48.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:48.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 110 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:48.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:48.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 111 2023-12-11T20:19:48.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:48.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 111 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:48.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:48.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 112 2023-12-11T20:19:48.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 10 client.4407.objecter checking op 18 2023-12-11T20:19:48.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 112 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:48.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:48.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:48.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:46.163+0000 7f50a77fe640 20 client.4407.objecter 18 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:19:48.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:47.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:48.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:47.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:48.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:47.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 24 ==== osd_op_reply(18 exist [writefull 0~5] v102'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(18 exist [writefull 0~5] v102'5 uv5 ondisk = 0) v8 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 18 ondisk uv 5 in 8.1 attempt 0 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 18 2023-12-11T20:19:48.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 18 2023-12-11T20:19:49.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:49.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 18 2023-12-11T20:19:49.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:19:49.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.909+0000 7f50b6b8a640 20 librados: complete_aio_write 0x55c136b844e0 2023-12-11T20:19:49.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (12934 ms) 2023-12-11T20:19:49.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-12-11T20:19:49.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 20 librados: queue_aio_write 0x55c1369c1480 completion 0x55c136a112a0 write_seq 5 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136b80cf0 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 112 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [writefull 0~5 in=5b] tid 19 osd.6 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 19 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 15 client.4407.objecter _send_op 19 to 8.1 on osd.6 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:19 8.1 8:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) v8 -- 0x55c136b83a80 con 0x55c136a11d60 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:48.910+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 25 ==== osd_map(113..113 src has 1..114) v4 ==== 628+0+0 (crc 0 0 0) 0x7f50b0029840 con 0x55c136a11d60 2023-12-11T20:19:49.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(113..113 src has 1..114) v4 2023-12-11T20:19:49.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [113,113] > 112 2023-12-11T20:19:49.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 113 2023-12-11T20:19:49.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 10 client.4407.objecter checking op 19 2023-12-11T20:19:49.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 113 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:49.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:49.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:19:49.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.014+0000 7f50a77fe640 20 client.4407.objecter 19 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:19:49.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:49.406 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:49.400+0000 10a54640 -1 osd.3 114 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:50.386 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:50.374+0000 10a54640 -1 osd.3 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:51.497 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:51.491+0000 10a54640 -1 osd.3 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:52.634 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:52.625+0000 10a54640 -1 osd.3 116 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:55.321 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:55.287+0000 10a54640 -1 osd.3 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:19:55.661 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:55.661 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:55.661 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:49.020+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:19:55.661 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:50.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:55.661 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:50.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:55.661 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:50.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:51.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:51.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:51.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:51.161+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:52.020+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:52.020+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:52.020+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:53.022+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:53.022+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:53.022+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:54.022+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:54.022+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:55.662 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:54.022+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.022+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.022+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.022+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 26 ==== osd_op_reply(19 exist [writefull 0~5] v113'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(19 exist [writefull 0~5] v113'6 uv6 ondisk = 0) v8 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 19 ondisk uv 6 in 8.1 attempt 0 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 19 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 19 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 19 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b6b8a640 20 librados: complete_aio_write 0x55c136a112a0 2023-12-11T20:19:55.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (6752 ms) 2023-12-11T20:19:55.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-12-11T20:19:55.826 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 20 librados: queue_aio_write 0x55c1369c1480 completion 0x55c136a112a0 write_seq 6 2023-12-11T20:19:55.826 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 10 client.4407.objecter _op_submit op 0x55c136b80cf0 2023-12-11T20:19:55.826 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 20 client.4407.objecter _calc_target epoch 113 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:55.826 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:55.826 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 10 client.4407.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [6,4] primary 6 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 20 client.4407.objecter _get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 10 client.4407.objecter _op_submit oid exist '@8' '@8' [writefull 0~5 in=5b] tid 20 osd.6 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 20 client.4407.objecter get_session s=0x55c136986140 osd=6 3 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 15 client.4407.objecter _session_op_assign 6 20 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 15 client.4407.objecter _send_op 20 to 8.1 on osd.6 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- osd_op(unknown.0.0:20 8.1 8:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) v8 -- 0x55c136b83a80 con 0x55c136a11d60 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.662+0000 7f50b9045780 5 client.4407.objecter 1 in flight 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.826+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 27 ==== osd_map(114..118 src has 1..119) v4 ==== 7700+0+0 (crc 0 0 0) 0x7f50b0029840 con 0x55c136a11d60 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.826+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(114..118 src has 1..119) v4 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [114,118] > 113 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 114 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:19:55.827 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 114 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:55.828 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:55.828 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 115 2023-12-11T20:19:55.828 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:19:55.828 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 115 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:55.828 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:19:55.828 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 116 2023-12-11T20:19:55.828 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:19:55.828 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 116 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:19:58.288 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:58.137+0000 10a54640 -1 osd.3 118 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:20:00.077 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:19:59.804+0000 10a54640 -1 osd.3 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:20:01.506 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:20:01.344+0000 10a54640 -1 osd.3 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4648.0:2 31.1e 31:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-11T20:20:02.720 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:20:02.469+0000 10a54640 -1 osd.3 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4485.0:11 33.5 33:a557efb1:::obj8:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70) 2023-12-11T20:20:03.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:03.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 117 2023-12-11T20:20:03.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:03.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 117 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:03.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:03.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 118 2023-12-11T20:20:03.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 118 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:55.827+0000 7f50a77fe640 20 client.4407.objecter 20 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:56.022+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:56.022+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:56.022+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:56.162+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:57.023+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:57.023+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:57.023+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:58.023+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:58.023+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:58.023+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:03.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:59.024+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:59.024+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:59.024+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:19:59.024+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:00.025+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:00.025+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:00.025+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:01.025+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:01.025+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:01.025+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:01.164+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:02.025+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:02.025+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:02.025+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:03.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:03.028+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:06.814 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:20:06.629+0000 10a54640 -1 osd.3 123 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:24 10.f 10:f40f2702:test-rados-api-smithi086-92348-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) 2023-12-11T20:20:06.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:03.028+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:06.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:03.028+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:06.975 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:04.031+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:04.031+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:04.031+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:05.031+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:05.031+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:05.031+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.031+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.031+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.031+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.169+0000 7f50b85ed640 10 client.4407.objecter tick 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.169+0000 7f50b85ed640 2 client.4407.objecter tid 20 on osd.6 is laggy 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.169+0000 7f50b85ed640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.169+0000 7f50b85ed640 10 monclient: _renew_subs 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.169+0000 7f50b85ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:20:06.976 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.169+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=119}) v3 -- 0x7f5088002f90 con 0x55c136a4c5c0 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.169+0000 7f50b85ed640 1 -- 172.21.15.86:0/1257650968 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- ping magic: 0 v1 -- 0x7f5088003da0 con 0x55c136a11d60 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 1 -- 172.21.15.86:0/1257650968 <== mon.0 v2:172.21.15.86:3300/0 25 ==== osd_map(119..127 src has 1..127) v4 ==== 12090+0+0 (secure 0 0 0) 0x7f50a80b4060 con 0x55c136a4c5c0 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_map(119..127 src has 1..127) v4 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map got epochs [119,127] > 118 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 119 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 119 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 120 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 120 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:06.977 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 121 2023-12-11T20:20:07.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:07.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 121 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 122 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 122 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 123 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 123 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 124 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 124 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:07.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 125 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 125 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 126 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 126 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 3 client.4407.objecter handle_osd_map decoding incremental epoch 127 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 10 client.4407.objecter checking op 20 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target epoch 127 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter dump_active .. 0 homeless 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:06.977+0000 7f50a77fe640 20 client.4407.objecter 20 8.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-12-11T20:20:07.031 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:07.031+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:07.032 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:07.031+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:07.854 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:20:07.716+0000 10a54640 -1 osd.3 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:24 10.f 10:f40f2702:test-rados-api-smithi086-92348-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) 2023-12-11T20:20:09.290 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:20:09.096+0000 10a54640 -1 osd.3 125 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:24 10.f 10:f40f2702:test-rados-api-smithi086-92348-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:07.031+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:08.034+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:08.034+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:08.034+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:09.043+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:09.043+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:09.043+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:09.043+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.043+0000 7f50a67fc640 10 monclient: tick 2023-12-11T20:20:10.065 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.043+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.043+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 <== osd.6 v2:172.21.15.133:6826/1611390427 28 ==== osd_op_reply(20 exist [writefull 0~5] v118'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f50b0025040 con 0x55c136a11d60 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 10 client.4407.objecter ms_dispatch 0x55c1369bb1b0 osd_op_reply(20 exist [writefull 0~5] v118'7 uv7 ondisk = 0) v8 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 10 client.4407.objecter in handle_osd_op_reply 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 7 client.4407.objecter handle_osd_op_reply 20 ondisk uv 7 in 8.1 attempt 0 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 10 client.4407.objecter op 0 rval 0 len 0 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 15 client.4407.objecter handle_osd_op_reply completed tid 20 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 15 client.4407.objecter _finish_op 20 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 20 client.4407.objecter put_session s=0x55c136986140 osd=6 4 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 15 client.4407.objecter _session_op_remove 6 20 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 5 client.4407.objecter 0 in flight 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b6b8a640 20 librados: complete_aio_write 0x55c136a112a0 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (14404 ms) 2023-12-11T20:20:10.066 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b9045780 10 librados: watch_flush enter 2023-12-11T20:20:10.067 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b9045780 10 librados: watch_flush exit 2023-12-11T20:20:10.067 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b9045780 10 client.4407.objecter close_session for osd.6 2023-12-11T20:20:10.067 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x55c136a11d60 msgr2=0x55c136ac3dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:20:10.067 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b9045780 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x55c136a11d60 0x55c136ac3dc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:20:10.067 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b9045780 20 client.4407.objecter put_session s=0x55c136986140 osd=6 1 2023-12-11T20:20:10.067 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b9045780 20 client.4407.objecter shutdown clearing up homeless session... 2023-12-11T20:20:10.067 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.066+0000 7f50b9045780 10 client.4407.objecter successfully canceled tick 2023-12-11T20:20:10.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f509806aa90 msgr2=0x7f509806cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:20:10.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f509806aa90 0x7f509806cf50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f50ac005ee0 tx=0x7f50ac005490 comp rx=0 tx=0).stop 2023-12-11T20:20:10.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 10 monclient: shutdown 2023-12-11T20:20:10.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:20:10.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 msgr2=0x55c136ad7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:20:10.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 --2- 172.21.15.86:0/1257650968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c136a4c5c0 0x55c136ad7330 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f50a8027050 tx=0x7f50a80113d0 comp rx=0 tx=0).stop 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b6b8a640 1 -- 172.21.15.86:0/1257650968 reap_dead start 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 shutdown_connections 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 >> 172.21.15.86:0/1257650968 conn(0x55c136a4f090 msgr2=0x55c136adec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 shutdown_connections 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 -- 172.21.15.86:0/1257650968 wait complete. 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.067+0000 7f50b9045780 1 librados: shutdown 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [----------] 12 tests from AsioRados (96808 ms total) 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [----------] Global test environment tear-down 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (134121 ms total) 2023-12-11T20:20:10.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_asio: [ PASSED ] 12 tests. 2023-12-11T20:20:19.033 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-12-11T20:20:19.033 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-12-11T20:20:19.033 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-12-11T20:20:19.033 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-12-11T20:20:19.033 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (142769 ms) 2023-12-11T20:20:19.034 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (142769 ms total) 2023-12-11T20:20:19.034 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: 2023-12-11T20:20:19.034 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-12-11T20:20:19.034 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (142769 ms total) 2023-12-11T20:20:19.034 INFO:tasks.workunit.client.0.smithi086.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-12-11T20:20:22.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: Running main() from gmock_main.cc 2023-12-11T20:20:22.013 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (51432 ms) 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (27811 ms) 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (6462 ms) 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-12-11T20:20:22.014 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (10486 ms) 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (2143 ms) 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (4998 ms) 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (2427 ms) 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (1687 ms) 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (2313 ms) 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (3393 ms) 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-12-11T20:20:22.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5018 ms) 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (3196 ms) 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (4316 ms) 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (2280 ms) 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (423 ms) 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (5111 ms) 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (133496 ms total) 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-12-11T20:20:22.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (145846 ms total) 2023-12-11T20:20:22.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: Running main() from gmock_main.cc 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [----------] Global test environment set-up. 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: handle_notify cookie 94046003853216 notify_id 261993005056 notifier_gid 4470 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (59590 ms) 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: Trying... 2023-12-11T20:20:22.900 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: handle_notify cookie 94046016600784 notify_id 515396075520 notifier_gid 4876 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: Waiting for 3.000000000s 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: Timed out. 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: Flushing... 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: Flushed... 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (86927 ms) 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (146517 ms total) 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [----------] Global test environment tear-down 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (146517 ms total) 2023-12-11T20:20:22.901 INFO:tasks.workunit.client.0.smithi086.stdout: watch_notify: [ PASSED ] 2 tests. 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (1000 ms) 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (70 ms) 2023-12-11T20:20:25.818 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-12-11T20:20:25.819 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (148617 ms) 2023-12-11T20:20:25.819 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (149687 ms total) 2023-12-11T20:20:25.819 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: 2023-12-11T20:20:25.819 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-12-11T20:20:25.819 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (149687 ms total) 2023-12-11T20:20:25.819 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: Running main() from gmock_main.cc 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [----------] Global test environment set-up. 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (4582 ms) 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (14068 ms) 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2023-12-11T20:20:25.862 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (130897 ms) 2023-12-11T20:20:25.863 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [----------] 3 tests from NeoRadosCmd (149547 ms total) 2023-12-11T20:20:25.863 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: 2023-12-11T20:20:25.863 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [----------] Global test environment tear-down 2023-12-11T20:20:25.863 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [==========] 3 tests from 1 test suite ran. (149547 ms total) 2023-12-11T20:20:25.863 INFO:tasks.workunit.client.0.smithi086.stdout: cmd: [ PASSED ] 3 tests. 2023-12-11T20:20:35.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: Running main() from gmock_main.cc 2023-12-11T20:20:35.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-12-11T20:20:35.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [----------] Global test environment set-up. 2023-12-11T20:20:35.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-12-11T20:20:35.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-12-11T20:20:35.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (854 ms) 2023-12-11T20:20:35.146 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (75 ms) 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (149329 ms) 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:19.935466+0000 mon.a [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 42 sec, osd.5 has slow ops) 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:20.753481+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi086-92511-6", "tierpool": "test-rados-api-smithi086-92511-7", "force_nonempty": "--force-nonempty" }]': finished 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:21.921719+0000 mon.b [INF] from='client.? 172.21.15.86:0/2070002087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi086-92329-16"}]: dispatch 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:22.237722+0000 mon.b [INF] from='client.? 172.21.15.86:0/2535629200' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi086-92511-6", "overlaypool": "test-rados-api-smithi086-92511-7"}]: dispatch 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:23.337920+0000 mon.a [WRN] Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:23.875913+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi086-92329-16"}]: dispatch 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:23.944076+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi086-92511-6", "overlaypool": "test-rados-api-smithi086-92511-7"}]: dispatch 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:24.292757+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi086-92807-6", "pg_num": 4}]': finished 2023-12-11T20:20:35.147 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:24.875205+0000 mon.b [INF] from='client.? 172.21.15.86:0/818636698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi086-92407-10"}]: dispatch 2023-12-11T20:20:35.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:25.491254+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi086-92407-10"}]: dispatch 2023-12-11T20:20:35.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:25.543143+0000 mon.c [INF] from='client.? 172.21.15.86:0/2453466049' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi086-92488-10"}]: dispatch 2023-12-11T20:20:35.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:25.974609+0000 mon.b [INF] from='client.? 172.21.15.86:0/2247033786' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi086-92807-4", "tierpool": "test-rados-api-smithi086-92807-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-12-11T20:20:35.148 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:25.981178+0000 mon.b [INF] from='client.? 172.21.15.86:0/2070002087' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi086-92329-16"}]: dispatch 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:27.283407+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": " api_list: [==========] Running 11 tests from 3 test suites. 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [----------] Global test environment set-up. 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [----------] 7 tests from LibRadosList 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosList.ListObjects (137821 ms) 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (4273 ms) 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-12-11T20:20:43.126 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: myset foo1,foo2,foo3 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo1 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo2 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo3 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: myset foo1,foo4,foo5 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo4 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo5 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo1 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: myset foo6,foo7 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo7 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo6 2023-12-11T20:20:43.127 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns1:foo4 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns1:foo5 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns2:foo7 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns2:foo6 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns1:foo1 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: :foo1 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: :foo2 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: :foo3 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (10718 ms) 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 1 0 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 10 0 2023-12-11T20:20:43.128 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 13 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 7 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 14 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 0 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 15 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 11 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 5 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 8 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 6 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 3 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 4 0 2023-12-11T20:20:43.129 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 12 0 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 9 0 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 2 0 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (4506 ms) 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: x cursor=MIN 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=1 cursor=5:02547ec2:::1:head 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=10 cursor=5:52ea6a34:::10:head 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=13 cursor=5:566253c9:::13:head 2023-12-11T20:20:43.130 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=7 cursor=5:5c6b0b28:::7:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=14 cursor=5:62a1935d:::14:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=0 cursor=5:6cac518f:::0:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=15 cursor=5:863748b0:::15:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=11 cursor=5:89d3ae78:::11:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=5 cursor=5:b29083e3:::5:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=8 cursor=5:bd63b0f1:::8:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=6 cursor=5:c4fdafeb:::6:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=3 cursor=5:cfc208b3:::3:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=4 cursor=5:d83876eb:::4:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=12 cursor=5:de5d7c5f:::12:head 2023-12-11T20:20:43.131 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=9 cursor=5:e960b815:::9:head 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > oid=2 cursor=5:f905c69b:::2:head 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: FIRST> seek to MIN oid=1 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=1 cursor=5:02547ec2:::1:head 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:02547ec2:::1:head 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:02547ec2:::1:head -> 1 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=10 cursor=5:52ea6a34:::10:head 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:52ea6a34:::10:head 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:52ea6a34:::10:head -> 10 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=13 cursor=5:566253c9:::13:head 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:566253c9:::13:head 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:566253c9:::13:head -> 13 2023-12-11T20:20:43.132 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=7 cursor=5:5c6b0b28:::7:head 2023-12-11T20:20:43.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:5c6b0b28:::7:head 2023-12-11T20:20:43.133 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:5c6b0b28:::7:head -> 7 2023-12-11T20:20:45.908 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=14 cursor=5:62a1935d:::1testprofile-LibRadosLockECPP_smithi086-92488-10"}]: dispatch 2023-12-11T20:20:45.908 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:27.724320+0000 mon.c [INF] from='client.? 172.21.15.86:0/4249861611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi086-92348-22"}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:27.996634+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi086-92807-4", "tierpool": "test-rados-api-smithi086-92807-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:28.014387+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi086-92329-16"}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:28.026205+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi086-92511-6", "overlaypool": "test-rados-api-smithi086-92511-7"}]': finished 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:28.398749+0000 mon.c [INF] from='client.? 172.21.15.86:0/484451299' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:29.392420+0000 client.admin [INF] onexx 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:29.427201+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi086-92348-22"}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:29.437138+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:30.482474+0000 mon.a [WRN] Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:33.387001+0000 mon.b [INF] from='client.? 172.21.15.86:0/2247033786' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi086-92807-4", "overlaypool": "test-rados-api-smithi086-92807-6"}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:33.701914+0000 mon.b [INF] from='client.? 172.21.15.86:0/818636698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi086-92407-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:33.920947+0000 mon.c [INF] from='client.? 172.21.15.86:0/2453466049' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi086-92488-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:34.035982+0000 mon.c [INF] from='client.? 172.21.15.86:0/4249861611' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi086-92348-22"}]: dispatch 2023-12-11T20:20:45.909 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:34.581912+0000 mon.c [INF] from='client.? 172.21.15.86:0/1812543875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi086-92306-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-11T20:20:45.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:35.522395+0000 mon.c [INF] from='client.? 172.21.15.86:0/484451299' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-11T20:20:45.910 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:35.762712+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi086-92807-4", "overlaypool": "test-rados-api-smithi086-92807-6"}]: dispatch 2023-12-11T20:20:46.430 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:35.807179+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosL4:head 2023-12-11T20:20:46.430 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:62a1935d:::14:head 2023-12-11T20:20:46.430 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:62a1935d:::14:head -> 14 2023-12-11T20:20:46.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=0 cursor=5:6cac518f:::0:head 2023-12-11T20:20:46.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:6cac518f:::0:head 2023-12-11T20:20:46.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:6cac518f:::0:head -> 0 2023-12-11T20:20:46.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=15 cursor=5:863748b0:::15:head 2023-12-11T20:20:46.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:863748b0:::15:head 2023-12-11T20:20:46.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:863748b0:::15:head -> 15 2023-12-11T20:20:46.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=11 cursor=5:89d3ae78:::11:head 2023-12-11T20:20:46.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:89d3ae78:::11:head 2023-12-11T20:20:46.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:89d3ae78:::11:head -> 11 2023-12-11T20:20:46.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=5 cursor=5:b29083e3:::5:head 2023-12-11T20:20:46.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:b29083e3:::5:head 2023-12-11T20:20:46.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:b29083e3:::5:head -> 5 2023-12-11T20:20:46.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=8 cursor=5:bd63b0f1:::8:head 2023-12-11T20:20:46.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:bd63b0f1:::8:head 2023-12-11T20:20:46.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:bd63b0f1:::8:head -> 8 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=6 cursor=5:c4fdafeb:::6:head 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:c4fdafeb:::6:head 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:c4fdafeb:::6:head -> 6 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=3 cursor=5:cfc208b3:::3:head 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:cfc208b3:::3:head 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:cfc208b3:::3:head -> 3 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=4 cursor=5:d83876eb:::4:head 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:d83876eb:::4:head 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:d83876eb:::4:head -> 4 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=12 cursor=5:de5d7c5f:::12:head 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:de5d7c5f:::12:head 2023-12-11T20:20:46.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:de5d7c5f:::12:head -> 12 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=9 cursor=5:e960b815:::9:head 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:e960b815:::9:head 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:e960b815:::9:head -> 9 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : oid=2 cursor=5:f905c69b:::2:head 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:f905c69b:::2:head 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:f905c69b:::2:head -> 2 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:566253c9:::13:head 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:566253c9:::13:head expected=5:566253c9:::13:head 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:566253c9:::13:head -> 13 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=13 expected=13 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:52ea6a34:::10:head 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:52ea6a34:::10:head expected=5:52ea6a34:::10:head 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:52ea6a34:::10:head -> 10 2023-12-11T20:20:46.434 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=10 expected=10 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:5c6b0b28:::7:head 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:5c6b0b28:::7:head expected=5:5c6b0b28:::7:head 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:5c6b0b28:::7:head -> 7 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=7 expected=7 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:d83876eb:::4:head 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:d83876eb:::4:head expected=5:d83876eb:::4:head 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:d83876eb:::4:head -> 4 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=4 expected=4 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:89d3ae78:::11:head 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:89d3ae78:::11:head expected=5:89d3ae78:::11:head 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:89d3ae78:::11:head -> 11 2023-12-11T20:20:46.435 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=11 expected=11 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:863748b0:::15:head 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:863748b0:::15:head expected=5:863748b0:::15:head 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:863748b0:::15:head -> 15 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=15 expected=15 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:6cac518f:::0:head 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:6cac518f:::0:head expected=5:6cac518f:::0:head 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:6cac518f:::0:head -> 0 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=0 expected=0 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:62a1935d:::14:head 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:62a1935d:::14:head expected=5:62a1935d:::14:head 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:62a1935d:::14:head -> 14 2023-12-11T20:20:46.436 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=14 expected=14 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:cfc208b3:::3:heockEC_smithi086-92407-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:35.973807+0000 mon.a [INF] from='client.? 172.21.15.86:0/1623484313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi086-92766-12"}]: dispatch 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:36.147642+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi086-92488-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:36.161047+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi086-92329-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:36.164118+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi086-92681-7"}]': finished 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:36.165827+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi086-92293-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:36.167697+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi086-92723-7"}]': finished 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:36.471942+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi086-92348-22"}]: dispatch 2023-12-11T20:20:46.646 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:36.482491+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi086-92306-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-11T20:20:46.647 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:36.592418+0000 mon.a [INF] from='client.? 172.21.15.86:0/1623484313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi086-92766-12"}]: dispatch 2023-12-11T20:20:46.647 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:37.242334+0000 mon.a [INF] from='client.? 172.21.15.86:0/1623484313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi086-92766-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-11T20:20:46.647 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:37.723456+0000 client.admin [INF] twoxx 2023-12-11T20:20:46.647 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:37.734037+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-11T20:20:46.647 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:37.731400+0000 mon.c [INF] from='client.? 172.21.15.86:0/313422031' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi086-92723-7"}]: dispatch 2023-12-11T20:20:46.647 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:37.778871+0000 mon.c [INF] from='client.? 172.21.15.86:0/4249861611' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi086-92348-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-11T20:20:46.647 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:37.856024+0000 mon.b [INF] from='client.? 172.21.15.86:0/2070002087' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi086-92329-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi086-92329-16"}]: dispatch 2023-12-11T20:20:46.647 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:38.313018+0000 mon.b [INF] from='client.? 172.21.15.86:0/3646310978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi086-92681-7"}]: dispatch 2023-12-11T20:20:47.237 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-1 api_stat: Running main() from gmock_main.cc 2023-12-11T20:20:47.237 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-12-11T20:20:47.237 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [----------] Global test environment set-up. 2023-12-11T20:20:47.237 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStat.Stat (29707 ms) 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (21257 ms) 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStat.StatNS (52330 ms) 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (33 ms) 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2744 ms) 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [----------] 5 tests from LibRadosStat (106072 ms total) 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: 2023-12-11T20:20:47.238 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (19473 ms) 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (4329 ms) 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1783 ms) 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1898 ms) 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (27483 ms total) 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [----------] Global test environment tear-down 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (171182 ms total) 2023-12-11T20:20:47.239 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat: [ PASSED ] 9 tests. 2023-12-11T20:20:47.452 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-12-11T20:20:47.452 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: seed 92723 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (64639 ms) 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (8673 ms) 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (17 ms) 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (32820 ms) 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (5094 ms) 2023-12-11T20:20:47.453 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (111243 ms total) 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (17634 ms) 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (2045 ms) 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1707 ms) 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (1007 ms) 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (22393 ms total) 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (171390 ms total) 2023-12-11T20:20:47.454 INFO:tasks.workunit.client.0.smithi086.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-12-11T20:20:49.014 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:05:02.122 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:49.030 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:05:02.134 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:49.293 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:05:02.401 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:49.947 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:05:03.055 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:49.953 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:05:03.061 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:50.111 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:03.180 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:50.125 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:03.193 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:50.386 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:05:03.494 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:50.419 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:03.487 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:20:50.870 INFO:tasks.workunit.client.0.smithi086.stdout:2-11T20:20:39.486223+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi086-92329-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi086-92329-16"}]: dispatch 2023-12-11T20:20:50.870 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.538203+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi086-92681-7"}]: dispatch 2023-12-11T20:20:50.870 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.697970+0000 mon.a [INF] from='client.? 172.21.15.86:0/3599757685' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi086-92980-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-11T20:20:50.870 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.703152+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi086-92511-6"}]': finished 2023-12-11T20:20:50.870 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.705531+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi086-92807-4", "overlaypool": "test-rados-api-smithi086-92807-6"}]': finished 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.707667+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi086-92407-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.709904+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi086-92488-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.712069+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi086-92306-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.966043+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi086-92723-7"}]: dispatch 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:39.979645+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi086-92348-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:40.523861+0000 mon.a [WRN] Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:41.626244+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:41.080756+0000 mon.b [INF] from='client.? 172.21.15.86:0/2535629200' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi086-92511-6", "tierpool": "test-rados-api-smithi086-92511-7"}]: dispatch 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:41.466787+0000 mon.b [INF] from='client.? 172.21.15.86:0/818636698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi086-92407-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi086-92407-10"}]: dispatch 2023-12-11T20:20:50.871 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:41.634024+0000 mon.c [INF] from='client.? 172.21.15.86:0/2453466049' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi086-92488-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi086-92488-10"}]: dispatch 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:41.684047+0000 mon.b [INF] from='client.? 172.21.15.86:0/2247033786' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi086-92807-6", "mode": "writeback"}]: dispatch 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:43.337351+0000 mon.a [INF] from='client.? 172.21.15.86:0/1623484313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi086-92766-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:43.340032+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi086-92681-7"}]': finished 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:43.342588+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi086-92723-7"}]': finished 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:43.346263+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi086-92348-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:43.558720+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi086-92488-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi086-92488-10"}]: dispatch 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:43.571398+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi086-92511-6", "tierpool": "test-rados-api-smithi086-92511-7"}]: dispatch 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:43.592623+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi086-92407-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi086-92407-10"}]: dispatch 2023-12-11T20:20:55.351 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:43.603194+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi086-92807-6", "mode": "writeback"}]: dispatch 2023-12-11T20:20:55.352 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:44.103230+0000 mon.a [INF] from='client.? 172.21.15.86:0/1623484313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi086-92766-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi086-92766-12"}]: dispatch 2023-12-11T20:20:55.352 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:44.441937+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-12-11T20:20:55.352 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:45.553604+0000 mon.a [WRN] Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-11T20:20:55.352 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:44.652617+0000 mon.c [INF] from='client.? 172.21.15.86:0/4249861611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi086-92348-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi086-92348-22"}]: dispatch 2023-12-11T20:20:55.352 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:45.529242+0000 mon.c [INF] from='client.? 172.21.15.86:0/484451299' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-11T20:20:55.352 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:47.182337+0000 client.admin [INF] threexx 2023-12-11T20:20:55.352 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:47.205787+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi086-92348-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi086-92348-22"}]: dispatch 2023-12-11T20:20:55.352 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:47.226235+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-11T20:20:55.613 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_1_[93095]: starting. 2023-12-11T20:20:55.613 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_1_[93095]: creating pool ceph_test_rados_list_parallel.smithi086-93031 2023-12-11T20:20:55.613 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_1_[93095]: created object 0... 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_1_[93095]: created object 25... 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_1_[93095]: created object 49... 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_1_[93095]: finishing. 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_1_[93095]: shutting down. 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_2_[93098]: starting. 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_2_[93098]: listing objects. 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_2_[93098]: listed object 0... 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_2_[93098]: listed object 25... 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_2_[93098]: saw 50 objects 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_2_[93098]: shutting down. 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_3_[116094]: starting. 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_3_[116094]: creating pool ceph_test_rados_list_parallel.smithi086-93031 2023-12-11T20:20:55.614 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_3_[116094]: created object 0... 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_3_[116094]: created object 25... 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_3_[116094]: created object 49... 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_3_[116094]: finishing. 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_3_[116094]: shutting down. 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_4_[116095]: starting. 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_4_[116095]: listing objects. 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_4_[116095]: listed object 0... 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_4_[116095]: listed object 25... 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_4_[116095]: saw 46 objects 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_4_[116095]: shutting down. 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_5_[116097]: starting. 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_5_[116097]: removed 25 objects... 2023-12-11T20:20:55.615 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_5_[116097]: removed half of the objects 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_5_[116097]: removed 50 objects... 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_5_[116097]: removed 50 objects 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_5_[116097]: shutting down. 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_6_[128058]: starting. 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_6_[128058]: creating pool ceph_test_rados_list_parallel.smithi086-93031 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_6_[128058]: created object 0... 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_6_[128058]: created object 25... 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_6_[128058]: created object 49... 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_6_[128058]: finishing. 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_6_[128058]: shutting down. 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.616 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_7_[128059]: starting. 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_7_[128059]: listing objects. 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_7_[128059]: listed object 0... 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_7_[128059]: listed object 25... 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_7_[128059]: listed object 50... 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_7_[128059]: saw 56 objects 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_7_[128059]: shutting down. 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_8_[128061]: starting. 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_8_[128061]: added 25 objects... 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_8_[128061]: added half of the objects 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_8_[128061]: added 50 objects... 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_8_[128061]: added 50 objects 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_8_[128061]: shutting down. 2023-12-11T20:20:55.617 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:55.618 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:20:57.328 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-12-11T20:20:57.328 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: notify 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (67542 ms) 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (44850 ms) 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (112392 ms total) 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: notify 2023-12-11T20:20:57.329 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (17319 ms) 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: notify 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (36029 ms) 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (233 ms) 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (1246 ms) 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894400368 notify_id 597000454145 notifier_gid 4948 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (315 ms) 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-12-11T20:20:57.330 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894391296 notify_id 597000454144 notifier_gid 4948 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (250 ms) 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894400368 notify_id 597000454146 notifier_gid 4948 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (281 ms) 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894400368 notify_id 597000454147 notifier_gid 4948 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (269 ms) 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894400368 notify_id 597000454148 notifier_gid 4948 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (341 ms) 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894400368 notify_id 597000454145 notifier_gid 4948 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (203 ms) 2023-12-11T20:20:57.331 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-12-11T20:20:57.332 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: trying... 2023-12-11T20:20:57.332 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894413216 notify_id 597000454149 notifier_gid 4948 2023-12-11T20:20:57.332 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: timed out 2023-12-11T20:21:07.644 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: flu open_pools_parallel: process_1_[93148]: starting. 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_1_[93148]: creating pool ceph_test_rados_open_pools_parallel.smithi086-93109 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_1_[93148]: created object 0... 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_1_[93148]: created object 25... 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_1_[93148]: created object 49... 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_1_[93148]: finishing. 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_1_[93148]: shutting down. 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_2_[93149]: starting. 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_2_[93149]: rados_pool_create. 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_2_[93149]: rados_ioctx_create. 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_2_[93149]: shutting down. 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: ******************************* 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_3_[119141]: starting. 2023-12-11T20:21:07.645 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_3_[119141]: creating pool ceph_test_rados_open_pools_parallel.smithi086-93109 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_3_[119141]: created object 0... 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_3_[119141]: created object 25... 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_3_[119141]: created object 49... 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_3_[119141]: finishing. 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_3_[119141]: shutting down. 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: ******************************* 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_4_[119142]: starting. 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_4_[119142]: rados_pool_create. 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_4_[119142]: rados_ioctx_create. 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: process_4_[119142]: shutting down. 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: ******************************* 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: ******************************* 2023-12-11T20:21:07.646 INFO:tasks.workunit.client.0.smithi086.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-12-11T20:21:07.783 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:47.243386+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi086-92329-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi086-92329-16"}]': finished 2023-12-11T20:21:07.783 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:47.246768+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi086-92511-6", "tierpool": "test-rados-api-smithi086-92511-7"}]': finished 2023-12-11T20:21:07.783 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: got: 2023-12-11T20:20:47.250132+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi086-92807-6", "mode": "writeback"}]': finished 2023-12-11T20:21:07.783 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (41424 ms) 2023-12-11T20:21:07.783 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (191682 ms total) 2023-12-11T20:21:07.783 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: 2023-12-11T20:21:07.783 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [----------] Global test environment tear-down 2023-12-11T20:21:07.783 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (191682 ms total) 2023-12-11T20:21:07.784 INFO:tasks.workunit.client.0.smithi086.stdout: api_cmd: [ PASSED ] 4 tests. 2023-12-11T20:21:10.550 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: Running main() from gmock_main.cc 2023-12-11T20:21:10.550 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-12-11T20:21:10.550 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [----------] Global test environment set-up. 2023-12-11T20:21:10.550 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-12-11T20:21:10.551 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: seed 92348 2023-12-11T20:21:10.551 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-12-11T20:21:10.551 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-12-11T20:21:10.551 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-12-11T20:21:10.551 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (66152 ms) 2023-12-11T20:21:10.551 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-12-11T20:21:10.552 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (22728 ms) 2023-12-11T20:21:10.552 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-12-11T20:21:10.552 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (3444 ms) 2023-12-11T20:21:10.552 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-12-11T20:21:10.552 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (2215 ms) 2023-12-11T20:21:10.552 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-12-11T20:21:10.553 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (1168 ms) 2023-12-11T20:21:10.553 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-12-11T20:21:10.553 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (35698 ms) 2023-12-11T20:21:10.553 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-12-11T20:21:10.553 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (1121 ms) 2023-12-11T20:21:10.553 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-12-11T20:21:10.554 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (1216 ms) 2023-12-11T20:21:10.554 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-12-11T20:21:10.554 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (169 ms) 2023-12-11T20:21:10.554 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-12-11T20:21:10.554 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (107 ms) 2023-12-11T20:21:10.554 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-12-11T20:21:10.554 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (159 ms) 2023-12-11T20:21:10.554 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (101 ms) 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2691 ms) 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (131 ms) 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (364 ms) 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-11T20:21:10.555 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (385 ms) 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (139 ms) 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (133 ms) 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (67 ms) 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (138188 ms total) 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: 2023-12-11T20:21:10.556 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-12-11T20:21:10.557 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-12-11T20:21:10.557 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (6443 ms) 2023-12-11T20:21:10.557 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-12-11T20:21:10.557 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (765 ms) 2023-12-11T20:21:10.557 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-12-11T20:21:10.557 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (219 ms) 2023-12-11T20:21:10.557 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-12-11T20:21:10.557 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (168 ms) 2023-12-11T20:21:10.558 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-12-11T20:21:13.078 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundT list_parallel: ******************************* 2023-12-11T20:21:13.078 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_9_[135969]: starting. 2023-12-11T20:21:13.078 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_9_[135969]: creating pool ceph_test_rados_list_parallel.smithi086-93031 2023-12-11T20:21:13.078 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_9_[135969]: created object 0... 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_9_[135969]: created object 25... 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_9_[135969]: created object 49... 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_9_[135969]: finishing. 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_9_[135969]: shutting down. 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_10_[135970]: starting. 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_10_[135970]: listing objects. 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_10_[135970]: listed object 0... 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_10_[135970]: listed object 25... 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_10_[135970]: listed object 50... 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_10_[135970]: listed object 75... 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_10_[135970]: saw 97 objects 2023-12-11T20:21:13.079 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_10_[135970]: shutting down. 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_13_[135973]: starting. 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_13_[135973]: removed 25 objects... 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_13_[135973]: removed half of the objects 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_13_[135973]: removed 50 objects... 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_13_[135973]: removed 50 objects 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_13_[135973]: shutting down. 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_12_[135972]: starting. 2023-12-11T20:21:13.080 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_12_[135972]: added 25 objects... 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_12_[135972]: added half of the objects 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_12_[135972]: added 50 objects... 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_12_[135972]: added 50 objects 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_12_[135972]: shutting down. 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_11_[135971]: starting. 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_11_[135971]: added 25 objects... 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_11_[135971]: added half of the objects 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_11_[135971]: added 50 objects... 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_11_[135971]: added 50 objects 2023-12-11T20:21:13.081 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_11_[135971]: shutting down. 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_14_[144753]: starting. 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_14_[144753]: creating pool ceph_test_rados_list_parallel.smithi086-93031 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_14_[144753]: created object 0... 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_14_[144753]: created object 25... 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_14_[144753]: created object 49... 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_14_[144753]: finishing. 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_14_[144753]: shutting down. 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.082 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:13.083 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: starting. 2023-12-11T20:21:13.083 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: listing objects. 2023-12-11T20:21:16.451 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: delete_pools_parallel: process_1_[93205]: starting. 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_1_[93205]: creating pool ceph_test_rados_delete_pools_parallel.smithi086-93157 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_1_[93205]: created object 0... 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_1_[93205]: created object 25... 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_1_[93205]: created object 49... 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_1_[93205]: finishing. 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_1_[93205]: shutting down. 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_2_[93207]: starting. 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_2_[93207]: deleting pool ceph_test_rados_delete_pools_parallel.smithi086-93157 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_2_[93207]: shutting down. 2023-12-11T20:21:16.452 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: ******************************* 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_3_[127538]: starting. 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_3_[127538]: creating pool ceph_test_rados_delete_pools_parallel.smithi086-93157 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_3_[127538]: created object 0... 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_3_[127538]: created object 25... 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_3_[127538]: created object 49... 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_3_[127538]: finishing. 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_3_[127538]: shutting down. 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: ******************************* 2023-12-11T20:21:16.453 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_5_[127540]: starting. 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_5_[127540]: listing objects. 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_5_[127540]: listed object 0... 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_5_[127540]: listed object 25... 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_5_[127540]: saw 50 objects 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_5_[127540]: shutting down. 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: ******************************* 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_4_[127539]: starting. 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_4_[127539]: deleting pool ceph_test_rados_delete_pools_parallel.smithi086-93157 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: process_4_[127539]: shutting down. 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: ******************************* 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: ******************************* 2023-12-11T20:21:16.454 INFO:tasks.workunit.client.0.smithi086.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-12-11T20:21:16.588 INFO:tasks.workunit.client.0.smithi086.stdout:cabc675590 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc675f90 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc676990 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc676fe0 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc6779c0 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc676790 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc6788b0 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc677710 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc679720 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc678620 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc67a240 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc67ac50 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc679540 2023-12-11T20:21:16.589 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc67bb40 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc67aa50 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc67ca70 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc67b8a0 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc483e30 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc4848c0 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc67c7f0 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc4857f0 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc484640 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc486680 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc485550 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc487510 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc486410 2023-12-11T20:21:16.590 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc488010 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc488ad0 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc487310 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc489960 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc4887e0 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabcfb12e0 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabcfb13f0 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabcfb1540 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabcfb16d0 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc4975e0 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc6775f0 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: waiting 0x55cabc488500 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: done. 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (51084 ms) 2023-12-11T20:21:16.591 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (51084 ms total) 2023-12-11T20:21:16.592 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: 2023-12-11T20:21:16.592 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-12-11T20:21:16.592 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-12-11T20:21:16.592 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-12-11T20:21:16.592 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-12-11T20:21:16.592 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMisc.Exec (55376 ms) 2023-12-11T20:21:16.592 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (4808 ms) 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2043 ms) 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4651 ms) 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-12-11T20:21:16.593 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (75300 ms) 2023-12-11T20:21:16.594 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] 8 tests from LibRadosMisc (142178 ms total) 2023-12-11T20:21:16.594 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: 2023-12-11T20:21:16.594 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [----------] Global test environment tear-down 2023-12-11T20:21:16.594 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (200594 ms total) 2023-12-11T20:21:16.594 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc: [ PASSED ] 12 tests. 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify_test_cb 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (99868 ms) 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-12-11T20:21:16.979 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94496255663168 err -107 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (10852 ms) 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94496255665488 err -107 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (9655 ms) 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_cb from 4627 notify_id 549755813889 cookie 94496255681952 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (9163 ms) 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_cb from 4627 notify_id 549755813889 cookie 94496255697984 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (402 ms) 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_cb from 4627 notify_id 549755813890 cookie 94496255697920 2023-12-11T20:21:16.980 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (253 ms) 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_cb from 4627 notify_id 549755813888 cookie 94496255697920 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_cb from 4627 notify_id 549755813888 cookie 94496255714912 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (357 ms) 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_cb from 4627 notify_id 549755813891 cookie 94496255697856 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_cb from 4627 notify_id 549755813892 cookie 94496255697856 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3220 ms) 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94496255697856 err -107 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_cb from 4627 notify_id 558345748482 cookie 94496255697856 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5274 ms) 2023-12-11T20:21:16.981 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94496255714864 err -107 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (2243 ms) 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (141287 ms total) 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: watch_notify_test_cb 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (14714 ms) 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (14715 ms total) 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-12-11T20:21:16.982 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (200902 ms total) 2023-12-11T20:21:16.983 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-12-11T20:21:18.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: Running main() from gmock_main.cc 2023-12-11T20:21:18.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-12-11T20:21:18.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [----------] Global test environment set-up. 2023-12-11T20:21:18.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-12-11T20:21:18.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-12-11T20:21:18.069 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (24655 ms) 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (32874 ms) 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (9735 ms) 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.Checksum (4040 ms) 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (2477 ms) 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (9971 ms) 2023-12-11T20:21:18.070 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6569 ms) 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (871 ms) 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.TruncTest (3275 ms) 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3080 ms) 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (1245 ms) 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.RmXattr (10008 ms) 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-12-11T20:21:18.071 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIo.XattrIter (23666 ms) 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [----------] 14 tests from LibRadosIo (132466 ms total) 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (9892 ms) 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (401 ms) 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (690 ms) 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (428 ms) 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (2999 ms) 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-12-11T20:21:18.072 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (163 ms) 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (935 ms) 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (1034 ms) 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (965 ms) 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (575 ms) 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [----------] 10 tests from LibRadosIoEC (18082 ms total) 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [----------] Global test environment tear-down 2023-12-11T20:21:18.073 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [==========] 24 tests from 2 test suites ran. (202171 ms total) 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: seed 92488 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (27078 ms) 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-12-11T20:21:18.074 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (3201 ms) 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (16774 ms) 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (24392 ms) 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (7624 ms) 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (37639 ms) 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (13742 ms) 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3221 ms) 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (133671 ms total) 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: 2023-12-11T20:21:18.075 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (10167 ms) 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (414 ms) 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (2169 ms) 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1247 ms) 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (298 ms) 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (1703 ms) 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (641 ms) 2023-12-11T20:21:18.076 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-12-11T20:21:18.077 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (158 ms) 2023-12-11T20:21:18.077 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (16797 ms total) 2023-12-11T20:21:18.077 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: 2023-12-11T20:21:18.077 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-12-11T20:21:18.077 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (202110 ms total) 2023-12-11T20:21:18.077 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-12-11T20:21:18.077 INFO:tasks.workunit.client.0.smithi086.stdout: api_io: [ PASSED ] 24 tests. 2023-12-11T20:21:18.248 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: Running main() from gmock_main.cc 2023-12-11T20:21:18.248 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-12-11T20:21:18.248 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [----------] Global test environment set-up. 2023-12-11T20:21:18.248 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (12626 ms) 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLock.LockShared (40341 ms) 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (25560 ms) 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (21857 ms) 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (30516 ms) 2023-12-11T20:21:18.249 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLock.Unlock (2005 ms) 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (292 ms) 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (161 ms) 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [----------] 8 tests from LibRadosLock (133358 ms total) 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (7628 ms) 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (183 ms) 2023-12-11T20:21:18.250 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (2003 ms) 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1165 ms) 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (250 ms) 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (1739 ms) 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (250 ms) 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (374 ms) 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (13592 ms total) 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [----------] Global test environment tear-down 2023-12-11T20:21:18.251 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (202309 ms total) 2023-12-11T20:21:18.252 INFO:tasks.workunit.client.0.smithi086.stdout: api_lock: [ PASSED ] 16 tests. 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout:shing 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: flushed 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3425 ms) 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: trying... 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894400368 notify_id 597000454150 notifier_gid 4948 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: timed out 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: flushing 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: flushed 2023-12-11T20:21:19.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3160 ms) 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: List watches 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: notify2 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894400368 notify_id 601295421447 notifier_gid 4948 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: notify2 done 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: watch_check 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: unwatch2 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: flushing 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: done 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3925 ms) 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: List watches 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: notify2 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: handle_notify cookie 94053894400368 notify_id 605590388738 notifier_gid 4948 2023-12-11T20:21:19.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: notify2 done 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: watch_check 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: unwatch2 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: flushing 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: done 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3321 ms) 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (70317 ms total) 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (202975 ms total) 2023-12-11T20:21:19.088 INFO:tasks.workunit.client.0.smithi086.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-12-11T20:21:21.812 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:05:34.920 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:21.860 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:05:34.967 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:21.901 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:34.969 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:21.912 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:34.981 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:21.964 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:05:35.073 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:21.988 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:35.056 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:22.026 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:05:35.134 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:22.037 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:05:35.146 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:22.152 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:05:35.260 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: listed object 0... 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: listed object 25... 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: listed object 50... 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: listed object 75... 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: listed object 100... 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: listed object 125... 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: saw 150 objects 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_15_[144754]: shutting down. 2023-12-11T20:21:36.184 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_16_[144755]: starting. 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_16_[144755]: added 25 objects... 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_16_[144755]: added half of the objects 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_16_[144755]: added 50 objects... 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_16_[144755]: added 50 objects 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: process_16_[144755]: shutting down. 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******************************* 2023-12-11T20:21:36.185 INFO:tasks.workunit.client.0.smithi086.stdout: list_parallel: ******* SUCCESS ********** 2023-12-11T20:21:46.165 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: Running main() from gmock_main.cc 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] Global test environment set-up. 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (37515 ms) 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (18116 ms) 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (9464 ms) 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5873 ms) 2023-12-11T20:21:46.166 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (70968 ms total) 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (13598 ms) 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (11017 ms) 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (15693 ms) 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (40308 ms total) 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-12-11T20:21:46.167 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (34837 ms) 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (20640 ms) 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (19652 ms) 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (5113 ms) 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (80242 ms total) 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-12-11T20:21:46.168 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (8389 ms) 2023-12-11T20:21:46.169 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-12-11T20:21:46.169 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (8436 ms) 2023-12-11T20:21:46.169 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (16825 ms total) 2023-12-11T20:21:46.169 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: 2023-12-11T20:21:46.169 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [----------] Global test environment tear-down 2023-12-11T20:21:46.169 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (230156 ms total) 2023-12-11T20:21:46.169 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-12-11T20:21:46.869 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:05:59.977 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:46.876 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:05:59.984 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:46.928 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:59.997 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:46.929 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:59.998 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:46.936 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:00.045 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:46.957 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:00.065 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:47.008 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:00.076 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:47.027 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:00.135 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:47.057 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:00.165 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:21:50.663 INFO:tasks.workunit.client.0.smithi086.stdout:ripPP2 (139 ms) 2023-12-11T20:21:50.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-12-11T20:21:50.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (1931 ms) 2023-12-11T20:21:50.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-12-11T20:21:50.663 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (507 ms) 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (98 ms) 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (242 ms) 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (143 ms) 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (174 ms) 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (9231 ms) 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (13729 ms) 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-12-11T20:21:50.664 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (3560 ms) 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (3048 ms) 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1298 ms) 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (2811 ms) 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (44506 ms total) 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [----------] Global test environment tear-down 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (234760 ms total) 2023-12-11T20:21:50.665 INFO:tasks.workunit.client.0.smithi086.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-12-11T20:22:07.180 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:20.288 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:07.182 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:20.291 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:07.301 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:20.409 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:07.302 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:20.411 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:07.308 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:20.377 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:07.315 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:20.384 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:07.345 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:20.453 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:07.406 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:20.474 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:07.438 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:20.547 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:10.454 INFO:tasks.workunit.client.0.smithi086.stdout: misc: Running main() from gmock_main.cc 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [==========] Running 12 tests from 1 test suite. 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [----------] Global test environment set-up. 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [----------] 12 tests from NeoRadosMisc 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.Version 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.Version (5857 ms) 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4900 ms) 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.LongName (132133 ms) 2023-12-11T20:22:10.455 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (29496 ms) 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (41740 ms) 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (5687 ms) 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.Exec (6762 ms) 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (7731 ms) 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (5880 ms) 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2023-12-11T20:22:10.456 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.BigObject (6649 ms) 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2132 ms) 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (5136 ms) 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [----------] 12 tests from NeoRadosMisc (254103 ms total) 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [----------] Global test environment tear-down 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [==========] 12 tests from 1 test suite ran. (254103 ms total) 2023-12-11T20:22:10.457 INFO:tasks.workunit.client.0.smithi086.stdout: misc: [ PASSED ] 12 tests. 2023-12-11T20:22:27.040 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:40.148 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:27.101 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:40.209 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:27.158 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:40.226 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:27.159 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:40.228 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:27.166 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:40.273 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:27.168 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:40.277 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:27.174 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:06:40.283 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:27.235 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:40.304 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:27.241 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:06:40.350 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.324 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:00.432 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.393 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:00.462 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.398 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:00.466 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.408 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:00.517 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.417 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:00.525 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.424 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:00.532 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.489 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:00.557 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.503 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:00.611 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:47.522 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:00.630 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: Running main() from gmock_main.cc 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [----------] Global test environment set-up. 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (21300 ms) 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (95336 ms) 2023-12-11T20:22:49.582 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (42594 ms) 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (40903 ms) 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (13989 ms) 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (214122 ms total) 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (7809 ms) 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (12356 ms) 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2023-12-11T20:22:49.583 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (20758 ms) 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (9054 ms) 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (8418 ms) 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi086-93422-11. 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: Waiting for snaps to purge. 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20700 ms) 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (79097 ms total) 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [----------] Global test environment tear-down 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (293219 ms total) 2023-12-11T20:22:49.584 INFO:tasks.workunit.client.0.smithi086.stdout: snapshots: [ PASSED ] 11 tests. 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: seed 92639 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (23363 ms) 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (43297 ms) 2023-12-11T20:22:53.904 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (80964 ms) 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (18650 ms) 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (32307 ms) 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (198581 ms total) 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (7172 ms) 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (10176 ms) 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-12-11T20:22:53.905 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (15953 ms) 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (6754 ms) 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (6092 ms) 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-83-gc56ad3ea/rpm/el9/BUILD/ceph-19.0.0-83-gc56ad3ea/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi086-92639-7 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: waiting for snaps to purge 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (20486 ms) 2023-12-11T20:22:53.906 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (66633 ms total) 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (4072 ms) 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2239 ms) 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3792 ms) 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1926 ms) 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (12029 ms total) 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: 2023-12-11T20:22:53.907 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] api_aio: Running main() from gmock_main.cc 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [----------] Global test environment set-up. 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6764 ms) 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (64378 ms) 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (11911 ms) 2023-12-11T20:23:06.775 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (58830 ms) 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (31578 ms) 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (25701 ms) 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (14432 ms) 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (7757 ms) 2023-12-11T20:23:06.776 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (6727 ms) 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (7738 ms) 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (5940 ms) 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (5515 ms) 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (8249 ms) 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (4955 ms) 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.Flush (4093 ms) 2023-12-11T20:23:06.777 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4894 ms) 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4584 ms) 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (5376 ms) 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2940 ms) 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (5052 ms) 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (4027 ms) 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4355 ms) 2023-12-11T20:23:06.778 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3498 ms) 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3150 ms) 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4013 ms) 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4431 ms) 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [----------] 26 tests from LibRadosAio (310889 ms total) 2023-12-11T20:23:06.779 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: 2023-12-11T20:23:06.780 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-12-11T20:23:07.376 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.Simple api_aio_pp: Running main() from gmock_main.cc 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6362 ms) 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (54910 ms) 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-12-11T20:23:07.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (84862 ms) 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (32096 ms) 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (23915 ms) 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (7235 ms) 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (6657 ms) 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (6372 ms) 2023-12-11T20:23:07.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (4625 ms) 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (4684 ms) 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3039 ms) 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (5816 ms) 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (6718 ms) 2023-12-11T20:23:07.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (8228 ms) 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3948 ms) 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (5138 ms) 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (5852 ms) 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3614 ms) 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-12-11T20:23:07.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (4297 ms) 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (4009 ms) 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3971 ms) 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3170 ms) 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (4169 ms) 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4267 ms) 2023-12-11T20:23:07.381 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-12-11T20:23:07.382 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (4493 ms) 2023-12-11T20:23:07.382 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-12-11T20:23:07.382 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3999 ms) 2023-12-11T20:23:07.382 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-12-11T20:23:07.383 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:20.492 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:07.441 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:20.508 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:07.444 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:20.513 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:07.453 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:20.561 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:07.512 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:20.620 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:07.515 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:20.583 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:07.520 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:20.629 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:07.523 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:20.632 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:07.590 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:20.698 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.197 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:25.305 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.199 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:25.308 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.223 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:25.331 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.260 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:25.328 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.262 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:25.330 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.267 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:25.375 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.272 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:25.380 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.288 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:25.356 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:12.316 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:25.424 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:13.619 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRados LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-12-11T20:23:13.619 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (6208 ms) 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (5634 ms) 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (5076 ms) 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (16918 ms total) 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (317576 ms total) 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-12-11T20:23:13.620 INFO:tasks.workunit.client.0.smithi086.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-11T20:23:27.559 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:40.667 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:27.562 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:40.670 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:27.622 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:40.730 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:27.624 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:40.732 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:27.627 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:07:40.735 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:27.643 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:40.711 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:27.653 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:40.721 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:27.685 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:07:40.793 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:27.698 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:40.767 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.759 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:00.867 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.763 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:00.871 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.771 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:00.839 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.778 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:00.846 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.823 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:00.931 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.836 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:00.944 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.842 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:00.948 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.843 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:00.911 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:47.912 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:01.020 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout:ad 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:cfc208b3:::3:head expected=5:cfc208b3:::3:head 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:cfc208b3:::3:head -> 3 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=3 expected=3 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:c4fdafeb:::6:head 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:c4fdafeb:::6:head expected=5:c4fdafeb:::6:head 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:c4fdafeb:::6:head -> 6 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=6 expected=6 2023-12-11T20:23:55.447 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:bd63b0f1:::8:head 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:bd63b0f1:::8:head expected=5:bd63b0f1:::8:head 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:bd63b0f1:::8:head -> 8 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=8 expected=8 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:b29083e3:::5:head 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:b29083e3:::5:head expected=5:b29083e3:::5:head 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:b29083e3:::5:head -> 5 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=5 expected=5 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:02547ec2:::1:head 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:02547ec2:::1:head expected=5:02547ec2:::1:head 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:02547ec2:::1:head -> 1 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=1 expected=1 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:f905c69b:::2:head 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:f905c69b:::2:head expected=5:f905c69b:::2:head 2023-12-11T20:23:55.448 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:f905c69b:::2:head -> 2 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=2 expected=2 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:e960b815:::9:head 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:e960b815:::9:head expected=5:e960b815:::9:head 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:e960b815:::9:head -> 9 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=9 expected=9 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : seek to 5:de5d7c5f:::12:head 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : cursor()=5:de5d7c5f:::12:head expected=5:de5d7c5f:::12:head 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: > 5:de5d7c5f:::12:head -> 12 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: : entry=12 expected=12 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (14272 ms) 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (139751 ms) 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: split 0/5 -> MIN 5:33333333::::head 2023-12-11T20:23:55.449 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: split 1/5 -> 5:33333333::::head 5:66666666::::head 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: split 2/5 -> 5:66666666::::head 5:99999999::::head 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: split 3/5 -> 5:99999999::::head 5:cccccccc::::head 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: split 4/5 -> 5:cccccccc::::head MAX 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (31537 ms) 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [----------] 7 tests from LibRadosList (342878 ms total) 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (3350 ms) 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: myset foo1,foo2,foo3 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo1 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo2 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo3 2023-12-11T20:23:55.450 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: myset foo1,foo4,foo5 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo4 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo5 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo1 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: myset foo6,foo7 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo7 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: foo6 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns1:foo4 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns1:foo5 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns2:foo7 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns2:foo6 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: ns1:foo1 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: :foo1 2023-12-11T20:23:55.451 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: :foo2 2023-12-11T20:23:55.452 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: :foo3 2023-12-11T20:23:55.452 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3236 ms) 2023-12-11T20:23:55.452 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-12-11T20:23:55.452 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 1 0 2023-12-11T20:23:55.452 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 10 0 2023-12-11T20:23:55.452 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 13 0 2023-12-11T20:24:06.085 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 7 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 14 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 0 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 15 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 11 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 5 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 8 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 6 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 3 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 4 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 12 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 9 0 2023-12-11T20:24:06.086 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 2 0 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (8329 ms) 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [----------] 3 tests from LibRadosListEC (14916 ms total) 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3218 ms) 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [----------] 1 test from LibRadosListNP (3218 ms total) 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [----------] Global test environment tear-down 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [==========] 11 tests from 3 test suites ran. (370150 ms total) 2023-12-11T20:24:06.087 INFO:tasks.workunit.client.0.smithi086.stdout: api_list: [ PASSED ] 11 tests. 2023-12-11T20:24:07.601 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:20.709 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:07.658 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:20.726 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:07.659 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:20.728 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:07.670 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:20.778 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:07.721 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:20.789 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:07.726 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:20.834 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:07.744 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:20.852 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:07.754 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:20.862 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:07.820 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:20.928 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.764 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:40.872 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.768 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:40.876 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.824 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:40.932 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.827 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:40.895 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.828 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:08:40.936 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.828 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:40.937 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.834 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:40.902 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.887 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:08:40.996 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:27.890 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:40.958 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:41.748 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: Running main() from gmock_main.cc 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [----------] Global test environment set-up. 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (59683 ms) 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (34002 ms) 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (11128 ms) 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (9490 ms) 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2023-12-11T20:24:41.749 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (19891 ms) 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (30693 ms) 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (240479 ms) 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (405366 ms total) 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [----------] Global test environment tear-down 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (405366 ms total) 2023-12-11T20:24:41.750 INFO:tasks.workunit.client.0.smithi086.stdout: write_operations: [ PASSED ] 7 tests. 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-12-11T20:24:44.152 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (27452 ms) 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (38704 ms) 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (22100 ms) 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (57444 ms) 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (26019 ms) 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (33886 ms) 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (202364 ms) 2023-12-11T20:24:44.153 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (407969 ms total) 2023-12-11T20:24:44.154 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: 2023-12-11T20:24:44.154 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-12-11T20:24:44.154 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (407969 ms total) 2023-12-11T20:24:44.154 INFO:tasks.workunit.client.0.smithi086.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-12-11T20:24:47.875 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:00.983 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:47.878 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:00.987 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:47.923 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:00.992 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:47.924 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:01.033 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:47.926 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:00.995 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:47.930 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:01.039 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:47.933 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:01.042 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:47.973 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:01.042 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:47.987 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:01.095 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:50.376 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-12-11T20:24:50.376 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-12-11T20:24:50.376 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-12-11T20:24:50.376 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-12-11T20:24:50.376 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-12-11T20:24:50.376 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: seed 92557 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (417 ms) 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (55763 ms) 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (33956 ms) 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (25768 ms) 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (14343 ms) 2023-12-11T20:24:50.377 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1047 ms) 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (270 ms) 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (2184 ms) 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (69 ms) 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (414 ms) 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (58 ms) 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (137 ms) 2023-12-11T20:24:50.378 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (205 ms) 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: osd_max_attr_size = 0 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (180629 ms) 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (10529 ms) 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: done waiting, doing copies 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: waiting for final deep scrubs... 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: done waiting 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (78782 ms) 2023-12-11T20:24:50.379 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (100 ms) 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (65 ms) 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5269 ms) 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-12-11T20:24:50.380 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-12-11T20:24:56.031 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:09.139 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:56.033 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:09.141 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:56.074 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:09.182 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:56.090 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:09.199 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:56.091 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:09.159 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:56.092 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:09.160 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:56.100 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:09.208 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:56.122 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:09.191 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:24:56.138 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:09.246 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:07.991 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:21.099 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:07.997 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:21.104 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:08.048 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:21.157 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:08.049 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:21.117 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:08.052 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:21.121 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:08.062 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:21.171 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:08.066 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:21.175 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:08.087 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:21.155 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:08.106 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:21.214 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:10.430 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [-----Write 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6774 ms) 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7624 ms) 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7467 ms) 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (8710 ms) 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (9963 ms) 2023-12-11T20:25:10.431 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8330 ms) 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7054 ms) 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7536 ms) 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7780 ms) 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (8046 ms) 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6795 ms) 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7757 ms) 2023-12-11T20:25:10.432 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7747 ms) 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7973 ms) 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6681 ms) 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7415 ms) 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (123653 ms total) 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [----------] Global test environment tear-down 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (434542 ms total) 2023-12-11T20:25:10.433 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio: [ PASSED ] 42 tests. 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: Running main() from gmock_main.cc 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [==========] Running 6 tests from 2 test suites. 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [----------] Global test environment set-up. 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [----------] 3 tests from NeoradosList 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ RUN ] NeoradosList.ListObjects 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ OK ] NeoradosList.ListObjects (147246 ms) 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ OK ] NeoradosList.ListObjectsNS (40400 ms) 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ OK ] NeoradosList.ListObjectsMany (173984 ms) 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: [----------] 3 tests from NeoradosList (361630 ms total) 2023-12-11T20:25:17.402 INFO:tasks.workunit.client.0.smithi086.stdout: list: 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [----------] 3 tests from NeoradosECList 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ RUN ] NeoradosECList.ListObjects 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ OK ] NeoradosECList.ListObjects (8071 ms) 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ RUN ] NeoradosECList.ListObjectsNS 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ OK ] NeoradosECList.ListObjectsNS (9904 ms) 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ RUN ] NeoradosECList.ListObjectsMany 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ OK ] NeoradosECList.ListObjectsMany (61464 ms) 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [----------] 3 tests from NeoradosECList (79439 ms total) 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [----------] Global test environment tear-down 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [==========] 6 tests from 2 test suites ran. (441070 ms total) 2023-12-11T20:25:17.403 INFO:tasks.workunit.client.0.smithi086.stdout: list: [ PASSED ] 6 tests. 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout:-----] 22 tests from LibRadosMiscPP (410005 ms total) 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2923 ms) 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2923 ms total) 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-12-11T20:25:19.695 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (727 ms) 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (43 ms) 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (770 ms total) 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (352 ms) 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (75 ms) 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (427 ms total) 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-12-11T20:25:19.696 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (709 ms) 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (81 ms) 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (790 ms total) 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1914 ms) 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1914 ms total) 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (443675 ms total) 2023-12-11T20:25:19.697 INFO:tasks.workunit.client.0.smithi086.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-12-11T20:25:28.102 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:41.208 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:28.102 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:41.210 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:28.143 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:41.211 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:28.145 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:09:41.254 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:28.147 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:41.216 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:28.148 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:41.257 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:28.151 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:41.260 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:28.192 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:41.260 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:28.203 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:09:41.312 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:45.806 INFO:tasks.workunit.client.0.smithi086.stdout:Aio.RoundTripAppendPP (5043 ms) 2023-12-11T20:25:45.806 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (4449 ms) 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2973 ms) 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3248 ms) 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: setting pool EIO 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: max_success 0, min_failed 1 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4511 ms) 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4444 ms) 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (331115 ms total) 2023-12-11T20:25:45.807 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3735 ms) 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10095 ms) 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (18285 ms) 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3594 ms) 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (35709 ms total) 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (4342 ms) 2023-12-11T20:25:45.808 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (4342 ms total) 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14053 ms) 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7152 ms) 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7951 ms) 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7941 ms) 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3062 ms) 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6895 ms) 2023-12-11T20:25:45.809 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7533 ms) 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6772 ms) 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7941 ms) 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6766 ms) 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7826 ms) 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7404 ms) 2023-12-11T20:25:45.810 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-12-11T20:25:45.811 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7456 ms) 2023-12-11T20:25:48.325 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:01.434 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:48.392 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:01.500 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:48.398 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:01.466 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:48.402 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:01.470 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:48.456 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:01.564 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:48.461 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:01.570 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:48.471 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:01.539 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:48.472 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:01.579 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:25:48.564 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:01.672 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.549 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:21.656 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.553 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:21.662 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.612 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:21.680 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.613 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:21.681 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.630 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:21.738 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.636 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:21.745 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.650 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:21.757 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.670 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:21.739 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:08.703 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:21.812 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:25.163 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 read_operations: Running main() from gmock_main.cc 2023-12-11T20:26:25.164 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2023-12-11T20:26:25.164 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [----------] Global test environment set-up. 2023-12-11T20:26:25.164 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2023-12-11T20:26:25.164 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2023-12-11T20:26:25.164 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (461884 ms) 2023-12-11T20:26:25.164 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2023-12-11T20:26:25.164 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (4228 ms) 2023-12-11T20:26:25.165 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2023-12-11T20:26:25.165 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3181 ms) 2023-12-11T20:26:25.165 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2023-12-11T20:26:25.165 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (4656 ms) 2023-12-11T20:26:25.165 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2023-12-11T20:26:25.165 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3249 ms) 2023-12-11T20:26:25.165 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2023-12-11T20:26:25.165 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3530 ms) 2023-12-11T20:26:25.166 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2023-12-11T20:26:25.166 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3348 ms) 2023-12-11T20:26:25.166 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2023-12-11T20:26:25.166 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3066 ms) 2023-12-11T20:26:25.166 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2023-12-11T20:26:25.166 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3243 ms) 2023-12-11T20:26:25.166 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2023-12-11T20:26:25.166 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3753 ms) 2023-12-11T20:26:25.167 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2023-12-11T20:26:25.167 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3242 ms) 2023-12-11T20:26:25.167 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2023-12-11T20:26:25.167 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (4652 ms) 2023-12-11T20:26:25.167 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2023-12-11T20:26:25.167 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3549 ms) 2023-12-11T20:26:25.167 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2023-12-11T20:26:25.168 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3219 ms) 2023-12-11T20:26:25.168 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (508800 ms total) 2023-12-11T20:26:25.168 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: 2023-12-11T20:26:25.168 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [----------] Global test environment tear-down 2023-12-11T20:26:25.168 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (508801 ms total) 2023-12-11T20:26:25.168 INFO:tasks.workunit.client.0.smithi086.stdout: read_operations: [ PASSED ] 14 tests. 2023-12-11T20:26:28.735 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:41.843 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:28.740 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:41.848 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:28.795 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:41.863 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:28.801 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:41.870 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:28.816 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:41.925 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:28.818 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:41.927 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:28.839 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:10:41.948 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:28.856 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:41.924 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:28.875 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:10:41.984 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:33.793 INFO:tasks.workunit.client.0.smithi086.stdout: 2023-12-11T20:26:33.793 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3599 ms) 2023-12-11T20:26:33.793 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-12-11T20:26:33.793 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7688 ms) 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7450 ms) 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7095 ms) 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7010 ms) 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7613 ms) 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7529 ms) 2023-12-11T20:26:33.794 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (146738 ms total) 2023-12-11T20:26:33.795 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: 2023-12-11T20:26:33.795 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-12-11T20:26:33.795 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (517904 ms total) 2023-12-11T20:26:33.795 INFO:tasks.workunit.client.0.smithi086.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-12-11T20:26:46.894 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:00.002 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:46.931 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:00.039 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:46.940 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:00.048 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:46.942 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:00.051 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:46.954 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:00.023 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:46.959 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:00.027 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:46.984 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:00.092 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:46.985 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:00.094 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:47.009 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:00.077 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.628 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:01.736 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.685 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:01.793 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.699 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:01.807 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.733 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:01.801 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.735 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:01.804 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.748 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:01.817 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.762 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:01.870 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.764 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:01.872 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:26:48.776 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:01.884 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.742 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:21.850 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.746 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:21.854 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.777 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:21.885 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.794 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:21.862 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.795 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:21.904 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.798 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:21.866 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.799 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:21.907 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.826 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:21.934 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:08.826 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:21.894 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:28.915 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:42.023 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:28.975 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:42.083 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:28.993 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:11:42.102 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:29.022 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:42.130 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:29.023 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:42.131 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:29.028 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:42.096 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:29.031 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:42.099 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:29.045 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:11:42.152 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:29.053 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:42.121 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.023 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:02.131 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.025 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:02.134 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.060 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:02.168 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.075 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:02.184 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.079 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:02.187 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.081 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:02.150 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.089 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:02.157 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.113 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:02.221 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:49.124 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:02.192 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:58.995 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:12.103 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:58.999 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:12.108 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:59.026 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:12.135 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:59.047 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:12.116 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:59.050 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:12.119 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:59.058 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:12.166 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:59.059 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:12.167 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:59.075 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:12.143 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:27:59.084 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:12.193 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.243 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:22.351 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.247 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:22.356 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.277 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:22.386 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.306 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:22.375 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.319 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:22.387 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.320 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:22.429 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.325 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:22.434 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.350 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:22.418 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:09.356 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:22.465 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.135 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:27.243 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.137 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:27.246 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.158 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:27.266 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.199 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:27.268 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.200 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:27.308 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.203 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:27.311 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.204 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:27.272 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.222 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:27.330 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:14.223 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:27.291 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.352 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:42.460 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.356 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:42.465 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.383 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:42.492 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.404 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:42.472 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.405 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:42.474 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.406 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:42.515 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.408 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:42.517 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.435 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:42.544 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:29.444 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:42.513 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:30.401 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-12-11T20:28:30.401 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-12-11T20:28:30.401 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-12-11T20:28:30.401 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: seed 92511 2023-12-11T20:28:30.401 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-12-11T20:28:30.401 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (69363 ms) 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (45640 ms) 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (1465 ms) 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (116468 ms total) 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (65304 ms) 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (54514 ms) 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (21832 ms) 2023-12-11T20:28:30.402 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: my_snaps [4] 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: my_snaps [5,4] 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: my_snaps [6,5,4] 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: promoting some heads 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: promoting from clones for snap 7 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: promoting from clones for snap 6 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: promoting from clones for snap 5 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: promoting from clones for snap 4 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: waiting for scrubs... 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: done waiting 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (192672 ms) 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-12-11T20:28:30.403 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (11459 ms) 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (11309 ms) 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (9749 ms) 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10383 ms) 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (13718 ms) 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10265 ms) 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (12382 ms) 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (38490 ms) 2023-12-11T20:28:30.404 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10024 ms) 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (9626 ms) 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (17574 ms) 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (10363 ms) 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-12-11T20:28:30.405 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11904 ms) 2023-12-11T20:28:34.187 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:47.296 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:34.232 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:47.300 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:34.234 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:47.342 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:34.237 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:47.304 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:34.253 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:47.322 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:34.257 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:47.365 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:34.297 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:47.405 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:34.300 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:47.408 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:34.315 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:47.424 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.209 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:57.317 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.260 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:57.368 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.283 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:12:57.392 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.303 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:57.371 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.305 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:57.374 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.311 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:57.420 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.315 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:57.424 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.329 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:57.398 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:44.341 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:12:57.450 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:53.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-12-11T20:28:53.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-11T20:28:53.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: ok, hit_set contains 266:602f83fe:::foo:head 2023-12-11T20:28:53.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10019 ms) 2023-12-11T20:28:53.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-12-11T20:28:53.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg_num = 32 2023-12-11T20:28:53.015 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 0 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 1 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 2 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 3 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 4 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 5 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 6 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 7 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 8 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 9 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 10 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 11 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 12 ls 1702326528,0 2023-12-11T20:28:53.016 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 13 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 14 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 15 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 16 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 17 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 18 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 19 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 20 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 21 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 22 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 23 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 24 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 25 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 26 ls 1702326528,0 2023-12-11T20:28:53.017 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 27 ls 1702326528,0 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 28 ls 1702326528,0 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 29 ls 1702326528,0 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 30 ls 1702326528,0 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg 31 ls 1702326528,0 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: pg_num = 32 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:6cac518f:::0:head 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:02547ec2:::1:head 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f905c69b:::2:head 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:cfc208b3:::3:head 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d83876eb:::4:head 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b29083e3:::5:head 2023-12-11T20:28:53.018 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c4fdafeb:::6:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:5c6b0b28:::7:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:bd63b0f1:::8:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:e960b815:::9:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:52ea6a34:::10:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:89d3ae78:::11:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:de5d7c5f:::12:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:566253c9:::13:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:62a1935d:::14:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:863748b0:::15:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:3958e169:::16:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:4d4dabf9:::17:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:8391935d:::18:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:28883081:::19:head 2023-12-11T20:28:53.019 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:69259c59:::20:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:4bdb80b7:::21:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a11c5d71:::22:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:271af37b:::23:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:95b121be:::24:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:58d1031b:::25:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:0a050783:::26:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c709704c:::27:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:cbe56eaf:::28:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:86b4b162:::29:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:70d89383:::30:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:dd450c7c:::31:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:6d5729b1:::32:head 2023-12-11T20:28:53.020 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c388f3fb:::33:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:56cfea31:::34:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:9dbc1bf7:::35:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:40b74ccd:::36:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:4d5aaf42:::37:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:920f362c:::38:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:6cc53222:::39:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:9cad833f:::40:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:1ea84d41:::41:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c4480ef6:::42:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a694361e:::43:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d1bd33e9:::44:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:ddc2cd5d:::45:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:2b782207:::46:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:7b187fca:::47:head 2023-12-11T20:28:53.021 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:90ecdf6f:::48:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a5ed95fe:::49:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:ea0eaa55:::50:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f33ef17b:::51:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a0d1b2f6:::52:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:60c5229e:::53:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:edcbc575:::54:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:102cf253:::55:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:efb7fb0b:::56:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:50d0a326:::57:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d4dc5daf:::58:head 2023-12-11T20:28:53.022 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:3a130462:::59:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:ec87ed71:::60:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d5bc9454:::61:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:3ddfe313:::62:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:7c2816b9:::63:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:47e00e4d:::64:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c6410c18:::65:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b48ed237:::66:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:cd63ad31:::67:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b179e92b:::68:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:0d9f741a:::69:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:6d3352ae:::70:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c6d5c19e:::71:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:bc4729c3:::72:head 2023-12-11T20:28:53.023 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:77e930b9:::73:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:0abeecfd:::74:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b7c37e15:::75:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b6378398:::76:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:02bd68de:::77:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:cc795d2d:::78:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:630d4fea:::79:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:e0d29ef5:::80:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:fd6f13d2:::81:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:606461d5:::82:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:eadbdc43:::83:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:8761d0bb:::84:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:9ef0186f:::85:head 2023-12-11T20:28:53.024 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:e0d41294:::86:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:961de695:::87:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:1423148f:::88:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:633a8fa2:::89:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a8653809:::90:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:3dac8b33:::91:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:35aad435:::92:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f6dcc343:::93:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:dbbdad87:::94:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:1cb48ce0:::95:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:03cd461c:::96:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:17a4ea99:::97:head 2023-12-11T20:28:53.025 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:9993c9a7:::98:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:6394211c:::99:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:94c7ae57:::100:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:6fdee5bb:::101:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:9a477fd1:::102:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:eb850916:::103:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:affc56b9:::104:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b42dc814:::105:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f319f8f0:::106:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:9a40b9de:::107:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:8b524f28:::108:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:e3de589f:::109:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:90f90a5b:::110:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a7b4f1d7:::111:head 2023-12-11T20:28:53.026 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:af51766e:::112:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b6f90bd1:::113:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:e0261208:::114:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c9569ef7:::115:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:61bebe50:::116:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:fe93412b:::117:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d3d38bee:::118:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:3100ba0c:::119:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d0560ada:::120:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f0ea8b35:::121:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:766f231a:::122:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a07a2582:::123:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:bd7c6b3a:::124:head 2023-12-11T20:28:53.027 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:fb2ddaff:::125:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:4408e1fe:::126:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:ee1df7a7:::127:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c3002909:::128:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:4f48ffa9:::129:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:edf38733:::130:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c08425c0:::131:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:5f902d98:::132:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:41ea2c93:::133:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:813cee13:::134:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:0131818d:::135:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:26ba5a85:::136:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:381b8a5a:::137:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:28797e47:::138:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:bfca7f22:::139:head 2023-12-11T20:28:53.028 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:36807075:::140:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:80b03975:::141:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:5c15709b:::142:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f39ea15e:::143:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:ea992956:::144:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:48887b1c:::145:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:9f24a9dd:::146:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:987f100b:::147:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d2dd3581:::148:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:7fed1808:::149:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c80b70e9:::150:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:85ed90f9:::151:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:36428b24:::152:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d044c34a:::153:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:7c18bf58:::154:head 2023-12-11T20:28:53.029 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d1c21232:::155:head 2023-12-11T20:28:53.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a7a3c575:::156:head 2023-12-11T20:28:53.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:87da0633:::157:head 2023-12-11T20:28:53.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d5ac3822:::158:head 2023-12-11T20:28:53.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:3f20522d:::159:head 2023-12-11T20:28:53.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:6ca26563:::160:head 2023-12-11T20:28:53.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:532ce135:::161:head 2023-12-11T20:28:53.030 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:c78863e6:::162:head 2023-12-11T20:28:54.323 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:07.432 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:54.326 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:07.435 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:54.350 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:07.455 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:54.374 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:07.442 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:54.375 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:07.484 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:54.378 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:07.446 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:54.378 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:07.487 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:54.398 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:07.466 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:54.405 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:07.513 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.515 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:12.623 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.517 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:12.626 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.534 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:12.643 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.564 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:12.632 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.565 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:12.634 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.567 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:12.675 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.568 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:12.677 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.582 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:12.651 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:28:59.586 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:12.695 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.478 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:27.587 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.532 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:27.601 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.533 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:27.641 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.535 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:27.604 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.539 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:27.648 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.541 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:27.650 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.560 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:27.628 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.560 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:27.669 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:14.566 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:27.674 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.450 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:42.558 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.454 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:42.562 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.481 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:42.589 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.498 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:42.566 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.500 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:42.569 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.523 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:42.591 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.524 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:42.632 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.527 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:42.636 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:29.551 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:42.659 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.420 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:47.529 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.466 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:47.575 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.489 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:47.597 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.511 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:47.620 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.514 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:47.582 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.515 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:47.623 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.516 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:47.584 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.531 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:47.640 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:34.532 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:47.601 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.696 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:57.804 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.758 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:57.867 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.792 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:13:57.900 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.824 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:57.932 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.824 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:57.892 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.825 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:57.934 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.834 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:57.903 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.863 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:13:57.932 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:44.864 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:13:57.972 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.574 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:07.682 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.626 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:07.735 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.648 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:07.757 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.676 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:07.784 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.678 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:07.787 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.679 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:07.747 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.681 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:07.750 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.700 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:07.769 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:29:54.704 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:07.812 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:7462ddf6:::163:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:5d165639:::164:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f43765fc:::165:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b4c720e9:::166:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:e694b040:::167:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:afa38db2:::168:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:77ba9f53:::169:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:87495034:::170:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:7c96bf0e:::171:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:dbe346cc:::172:head 2023-12-11T20:30:04.157 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:e943ec24:::173:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f97a9c0c:::174:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:6f26e74d:::175:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:4f95e106:::176:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:0e6f2f8f:::177:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:05db05f1:::178:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:38a78d66:::179:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:d095610b:::180:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:a1a9d709:::181:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:1e5d39db:::182:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:f7df4fb9:::183:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:03a7f161:::184:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:ba70721e:::185:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:28e5662d:::186:head 2023-12-11T20:30:04.158 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:973d52de:::187:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:4303eb1c:::188:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:b990b48e:::189:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:29b8165b:::190:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:3547f197:::191:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:7e260936:::192:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:1abec7b1:::193:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:10fdda93:::194:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:15817eea:::195:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:770bab57:::196:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:ed9e13e7:::197:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:71471a8f:::198:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: checking for 267:10fb1d02:::199:head 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21697 ms) 2023-12-11T20:30:04.159 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: first is 1702326551 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,1702326556,1702326557,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,1702326556,1702326557,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,1702326556,1702326557,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,1702326556,1702326557,1702326559,1702326560,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,1702326556,1702326557,1702326559,1702326560,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326551,1702326553,1702326554,1702326556,1702326557,1702326559,1702326560,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702326554,1702326556,1702326557,1702326559,1702326560,1702326562,1702326563,0 2023-12-11T20:30:04.160 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: first now 1702326554, trimmed 2023-12-11T20:30:04.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22352 ms) 2023-12-11T20:30:04.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-12-11T20:30:04.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: foo0 2023-12-11T20:30:04.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-11T20:30:04.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (18362 ms) 2023-12-11T20:30:04.161 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-12-11T20:30:14.698 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:27.806 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:14.700 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:27.808 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:14.748 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:27.816 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:14.749 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:27.818 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:14.750 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:27.859 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:14.766 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:27.874 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:14.769 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:27.878 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:14.779 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:27.848 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:14.803 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:27.912 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.704 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:28.772 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.705 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:28.773 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.713 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:28.782 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.746 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:28.855 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.747 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:28.856 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.756 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:28.865 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.821 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:28.929 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.822 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:28.931 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:15.831 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:28.940 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.825 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:42.933 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.881 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:42.990 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.893 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:42.959 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.906 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:43.014 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.909 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:42.978 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.933 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:43.002 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.953 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:43.061 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.964 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:43.072 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:29.999 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:43.108 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.769 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:47.878 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.772 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:47.881 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.788 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:47.896 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.813 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:47.882 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.816 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:47.884 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.816 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:47.925 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.819 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:47.928 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.830 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:47.899 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:34.834 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:47.943 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:44.933 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:58.041 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:44.936 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:58.045 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:44.960 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:14:58.069 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:44.985 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:58.094 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:44.989 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:58.098 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:44.990 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:58.058 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:44.992 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:58.061 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:45.014 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:14:58.122 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:45.015 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:14:58.084 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.873 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:12.981 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.875 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:12.984 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.886 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:12.955 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.888 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:12.957 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.906 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:13.014 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.921 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:12.990 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.935 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:13.044 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.938 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:13.046 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:30:59.969 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:13.077 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.710 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:13.779 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.711 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:13.780 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.719 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:13.788 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.831 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:13.939 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.831 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:13.940 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.832 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:13.940 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.832 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:13.941 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.839 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:13.948 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:00.840 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:13.949 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.877 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:14.986 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.880 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:14.989 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.882 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:14.991 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.886 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:14.995 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.887 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:14.956 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.889 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:14.958 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.897 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:15.005 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.901 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:15.009 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:01.902 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:14.971 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.077 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:33.185 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.082 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:33.191 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.090 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:33.199 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.094 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:33.163 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.095 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:33.164 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.137 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:33.243 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.153 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:33.261 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.164 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:33.272 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:20.221 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:33.290 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.087 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:48.195 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.090 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:48.198 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.095 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:48.204 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.097 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:48.206 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.100 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:48.169 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.166 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:48.235 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.173 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:48.282 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.180 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:48.288 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:35.198 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:48.266 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.031 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:53.135 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.032 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:53.140 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.044 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:15:53.153 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.075 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:53.144 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.077 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:53.146 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.078 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:53.187 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.079 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:53.188 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.089 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:15:53.158 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:40.093 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:15:53.201 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.200 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:03.308 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.202 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:03.311 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.217 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:03.326 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.248 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:03.357 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.250 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:03.359 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.257 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:03.326 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.258 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:03.327 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.265 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:03.374 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:31:50.276 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:03.345 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.244 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:13.352 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.305 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:13.373 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.306 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:13.375 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.313 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:13.421 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.337 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:13.406 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.348 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:13.456 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.470 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:13.578 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.476 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:13.584 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:00.511 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:13.620 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.209 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:18.317 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.266 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:18.374 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.287 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:18.396 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.312 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:18.421 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.318 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:18.426 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.320 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:18.389 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.326 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:18.394 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.332 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:18.441 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:05.340 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:18.408 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.272 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:33.380 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.276 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:33.385 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.296 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:33.405 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.325 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:33.433 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.327 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:33.395 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.328 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:33.437 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.328 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:33.397 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.346 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:33.415 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:20.347 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:33.456 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.290 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:38.399 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.292 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:38.400 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.301 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:38.410 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.335 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:38.444 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.338 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:38.446 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.338 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:38.407 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.339 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:38.408 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.349 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:38.458 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:25.350 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:38.418 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.437 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:53.543 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.438 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:53.545 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.460 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:53.569 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.483 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:53.552 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.485 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:53.554 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.486 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:53.595 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.487 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:53.596 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.506 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:53.575 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:40.509 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:53.618 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.238 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:54.307 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.239 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:54.308 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.246 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:16:54.315 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.273 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:54.381 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.274 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:54.382 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.282 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:16:54.391 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.296 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:54.404 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.297 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:54.406 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:41.305 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:16:54.414 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.435 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:08.543 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.487 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:08.556 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.488 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:08.557 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.491 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:08.599 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.525 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:08.594 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.535 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:08.644 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.557 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:08.665 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.565 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:08.674 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:32:55.599 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:08.708 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.621 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:13.729 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.682 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:13.790 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.684 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:13.793 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.709 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:13.817 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.742 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:13.810 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.751 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:13.859 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.760 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:13.829 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.779 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:13.848 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:00.781 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:13.890 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.479 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:23.587 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.527 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:23.635 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.544 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:23.653 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.574 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:23.682 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.576 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:23.644 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.576 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:23.685 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.578 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:23.646 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.591 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:23.700 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:10.595 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:23.663 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.547 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:33.656 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.554 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:33.661 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.567 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:33.675 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.599 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:33.668 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.600 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:33.708 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.602 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:33.671 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.605 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:33.713 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.615 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:33.684 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:20.618 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:33.726 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:23.756 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:33:23.745+0000 24681640 -1 snap_mapper.add_oid found existing snaps mapped on 285:39c69ce4:test-rados-api-smithi086-92511-80::foo:2, removing 2023-12-11T20:33:23.767 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:33:23.762+0000 2c691640 -1 snap_mapper.add_oid found existing snaps mapped on 285:39c69ce4:test-rados-api-smithi086-92511-80::foo:2, removing 2023-12-11T20:33:30.591 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:43.699 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:30.592 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:43.701 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:30.614 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:43.723 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:30.638 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:43.707 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:30.641 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:43.710 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:30.642 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:43.750 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:30.644 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:43.753 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:30.660 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:43.729 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:30.664 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:43.773 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.186 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:45.294 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.192 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:45.300 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.211 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:45.319 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.255 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:45.324 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.257 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:45.326 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.270 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:45.338 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.287 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:45.395 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.287 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:45.396 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:32.301 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:45.410 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.620 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:53.728 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.621 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:53.730 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.640 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:53.749 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.669 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:53.737 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.670 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:53.738 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.670 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:53.779 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.672 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:53.781 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.688 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:53.757 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:40.692 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:53.800 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.634 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:58.742 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.636 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:58.744 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.652 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:17:58.760 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.681 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:58.750 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.682 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:58.751 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.696 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:17:58.765 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.698 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:58.807 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.703 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:58.811 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:33:45.718 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:17:58.826 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.859 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:13.967 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.867 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:13.976 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.908 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:13.977 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.913 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:13.982 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.927 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:14.035 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.929 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:14.038 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.945 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:14.053 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.948 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:14.017 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:00.970 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:14.079 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:01.942 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21326 ms) 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25001 ms) 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4003 ms) 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4507 ms) 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4526 ms) 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5421 ms) 2023-12-11T20:34:01.943 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5431 ms) 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (42134 ms) 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18288 ms) 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4441 ms) 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.944 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4733 ms) 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26534 ms) 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (17113 ms) 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6576 ms) 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5591 ms) 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-12-11T20:34:01.945 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: waiting for scrubs... 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: done waiting 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26705 ms) 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12009 ms) 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11270 ms) 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11181 ms) 2023-12-11T20:34:01.946 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-12-11T20:34:10.975 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:24.083 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:11.029 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:24.137 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:11.076 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:24.184 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:11.097 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:24.166 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:11.103 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:24.171 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:11.134 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:24.242 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:11.137 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:24.246 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:11.164 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:24.232 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:11.174 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:24.282 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:15.974 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:29.083 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:15.977 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:29.085 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:15.991 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:29.099 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:16.023 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:29.091 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:16.024 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:29.093 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:16.026 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:29.134 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:16.027 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:29.136 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:16.040 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:29.108 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:16.041 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:29.149 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.051 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:34.160 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.056 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:34.165 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.081 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:34.190 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.084 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:34.193 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.114 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:34.183 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.119 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:34.188 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.120 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:34.228 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.140 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:34.208 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:21.140 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:34.249 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.172 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:44.280 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.175 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:44.284 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.198 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:44.307 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.220 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:44.289 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.221 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:44.290 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.228 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:44.337 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.233 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:44.342 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.243 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:44.312 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:31.258 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:44.366 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.261 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:59.369 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.262 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:59.371 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.281 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:18:59.389 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.308 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:59.376 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.310 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:59.419 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.312 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:59.381 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.317 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:59.425 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.329 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:18:59.398 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:34:46.333 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:18:59.442 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.368 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:14.477 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.371 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:14.480 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.393 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:14.501 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.418 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:14.487 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.419 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:14.528 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.420 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:14.529 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.421 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:14.490 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.443 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:14.551 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:01.454 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:14.522 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.215 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:19.323 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.222 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:19.331 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.234 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:19.343 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.270 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:19.378 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.271 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:19.340 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.273 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:19.382 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.275 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:19.343 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.283 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:19.392 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:06.287 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:19.355 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:07.256 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:35:07.253+0000 23e80640 -1 snap_mapper.add_oid found existing snaps mapped on 86:4b47357e:test-rados-api-smithi086-92511-97::foo:31, removing 2023-12-11T20:35:07.278 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:35:07.273+0000 2b68f640 -1 snap_mapper.add_oid found existing snaps mapped on 86:4b47357e:test-rados-api-smithi086-92511-97::foo:31, removing 2023-12-11T20:35:16.252 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:29.360 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:16.254 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:29.362 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:16.277 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:29.386 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:16.303 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:29.371 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:16.304 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:29.412 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:16.306 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:29.415 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:16.306 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:29.375 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:16.328 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:29.437 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:16.329 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:29.397 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.244 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:39.352 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.290 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:39.358 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.291 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:39.400 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.293 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:39.361 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.314 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:39.383 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.328 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:39.436 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.348 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:39.457 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.353 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:39.462 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:26.377 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:39.486 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.365 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:44.473 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.411 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:44.479 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.412 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:44.480 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.423 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:44.531 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.427 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:44.536 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.427 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:44.536 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.429 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:44.498 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.443 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:44.552 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:31.449 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:44.558 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.653 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:49.761 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.655 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:49.763 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.671 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:49.780 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.710 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:49.779 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.711 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:49.819 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.711 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:49.780 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.714 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:49.823 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.732 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:49.801 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:36.733 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:49.841 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.452 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:54.560 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.457 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:54.566 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.484 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:54.592 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.515 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:54.624 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.518 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:54.627 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.520 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:54.588 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.526 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:54.594 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.549 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:54.657 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:41.551 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:54.620 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.532 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:59.640 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.593 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:59.661 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.596 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:59.665 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.598 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:59.707 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.621 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:19:59.730 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.623 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:19:59.691 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.668 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:59.777 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.675 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:59.783 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:35:46.699 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:19:59.807 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.549 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:14.658 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.553 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:14.662 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.585 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:14.693 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.592 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:14.701 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.598 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:14.666 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.598 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:14.706 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.598 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:14.667 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.629 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:14.698 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:01.631 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:14.740 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.672 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:19.780 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.678 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:19.786 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.695 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:19.803 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.724 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:19.832 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.725 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:19.833 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.741 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:19.810 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.743 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:19.852 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.746 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:19.814 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:06.762 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:19.830 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.743 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:29.852 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.747 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:29.856 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.771 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:29.880 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.794 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:29.862 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.795 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:29.864 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.798 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:29.907 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.808 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:29.916 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.820 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:29.889 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:16.843 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:29.951 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.698 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:39.807 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.704 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:39.813 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.732 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:39.841 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.746 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:39.815 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.750 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:39.819 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.751 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:39.860 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.755 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:39.864 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.775 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:39.843 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:26.779 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:39.888 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:31.906 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:45.015 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:31.954 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:45.023 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:31.957 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:45.026 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:31.960 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:45.068 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:31.998 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:45.066 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:32.005 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:45.114 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:32.016 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:45.125 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:32.020 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:45.129 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:32.037 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:45.146 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.784 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:59.892 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.786 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:59.895 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.812 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:20:59.921 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.832 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:59.941 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.834 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:59.903 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.835 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:59.904 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.836 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:59.944 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.860 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:20:59.929 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:46.862 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:20:59.971 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.791 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:04.899 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.795 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:04.904 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.808 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:04.917 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.842 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:04.951 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.843 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:04.912 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.843 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:04.952 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.847 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:04.916 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.856 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:04.964 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:36:51.858 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:04.927 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:06.909 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:20.018 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:06.971 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:20.040 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:06.982 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:20.051 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:07.000 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:20.108 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:07.017 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:20.085 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:07.029 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:20.137 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:07.046 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:20.155 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:07.049 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:20.158 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:07.078 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:20.187 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:11.979 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:25.087 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:11.985 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:25.094 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:12.001 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:25.110 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:12.032 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:25.140 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:12.033 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:25.142 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:12.043 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:25.112 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:12.044 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:25.113 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:12.052 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:25.161 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:12.060 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:25.129 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.057 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:35.165 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.061 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:35.169 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.084 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:35.192 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.108 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:35.216 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.109 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:35.177 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.109 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:35.218 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.110 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:35.179 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.133 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:35.202 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:22.136 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:35.245 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:31.965 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:45.073 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:31.967 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:45.076 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:31.989 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:45.098 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.012 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:45.120 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.015 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:45.083 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.015 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:45.124 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.016 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:45.085 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.038 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:45.147 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.039 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:45.108 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.826 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:45.895 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.827 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:45.896 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.840 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:45.909 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.873 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:45.981 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.873 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:45.982 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.893 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:46.002 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.911 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:46.020 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.913 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:46.021 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:32.927 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:46.036 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.055 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:55.163 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.058 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:55.167 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.094 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:21:55.202 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.108 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:55.177 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.111 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:55.180 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.119 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:55.228 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.122 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:55.231 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.133 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:21:55.202 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:42.145 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:21:55.254 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.194 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:05.302 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.198 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:05.306 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.225 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:05.333 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.253 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:05.361 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.254 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:05.363 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.258 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:05.327 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.259 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:05.328 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.279 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:05.388 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:52.283 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:05.351 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:55.911 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11074 ms) 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7976 ms) 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6614 ms) 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (28083 ms) 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:37:55.912 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15199 ms) 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12588 ms) 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: 00000010 69 73 |is| 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: 00000012 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12032 ms) 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: require_osd_release = reef 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6211 ms) 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (940566 ms total) 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-12-11T20:37:55.913 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1533 ms) 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11098 ms) 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18892 ms) 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (18 ms) 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31541 ms total) 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8294 ms) 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9927 ms) 2023-12-11T20:37:55.914 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: waiting for scrub... 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: done waiting 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26205 ms) 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10231 ms) 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8859 ms) 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9147 ms) 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12200 ms) 2023-12-11T20:37:55.915 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-12-11T20:37:57.278 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:10.386 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:57.341 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:10.410 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:57.342 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:10.411 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:57.344 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:10.452 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:57.345 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:10.454 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:57.353 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:10.461 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:57.368 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:10.476 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:57.370 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:10.439 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:37:57.387 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:10.495 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.418 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:25.526 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.420 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:25.528 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.447 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:25.555 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.462 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:25.531 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.465 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:25.534 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.466 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:25.575 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.468 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:25.577 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.491 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:25.560 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:12.506 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:25.615 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.357 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:40.466 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.360 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:40.469 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.391 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:40.500 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.410 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:40.519 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.412 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:40.481 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.413 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:40.522 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.415 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:40.484 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.442 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:40.551 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:27.448 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:40.517 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.363 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:45.471 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.366 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:45.475 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.383 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:45.492 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.413 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:45.482 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.414 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:45.523 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.415 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:45.483 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.418 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:45.527 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.430 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:45.499 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:32.434 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:45.543 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.416 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:55.525 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.420 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:55.528 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.447 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:22:55.555 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.465 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:55.534 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.469 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:55.537 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.492 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:22:55.561 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.497 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:55.606 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.500 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:55.609 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:42.526 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:22:55.634 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.449 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:05.557 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.453 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:05.561 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.479 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:05.588 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.500 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:05.568 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.503 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:05.572 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.518 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:05.627 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.523 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:05.632 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.526 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:05.595 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:38:52.548 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:05.657 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.686 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:15.795 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.689 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:15.798 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.721 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:15.829 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.747 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:15.816 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.749 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:15.817 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.757 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:15.866 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.758 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:15.867 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.773 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:15.842 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:02.784 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:15.893 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.572 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:25.681 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.577 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:25.686 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.602 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:25.711 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.621 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:25.690 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.624 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:25.693 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.639 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:25.747 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.644 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:25.753 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.646 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:25.714 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:12.670 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:25.778 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.587 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:30.695 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.591 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:30.700 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.603 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:30.712 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.633 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:30.741 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.633 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:30.702 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.637 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:30.746 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.638 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:30.706 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.648 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:30.717 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:17.649 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:30.757 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.652 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:45.760 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.666 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:45.774 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.693 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:45.801 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.698 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:45.806 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.701 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:45.810 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.711 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:45.780 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.713 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:45.782 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.735 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:45.844 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:32.736 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:45.805 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.602 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:46.670 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.603 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:46.671 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.615 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:23:46.684 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.634 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:46.743 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.635 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:46.744 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.648 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:23:46.757 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.737 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:46.846 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.738 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:46.847 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:33.752 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:23:46.861 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.710 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:00.818 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.714 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:00.823 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.745 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:00.854 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.759 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:00.827 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.762 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:00.870 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.763 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:00.832 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.764 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:00.873 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.793 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:00.861 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:47.794 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:00.903 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.726 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:05.834 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.728 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:05.837 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.754 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:05.863 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.774 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:05.882 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.775 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:05.843 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.776 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:05.845 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.777 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:05.885 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.786 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:05.855 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:39:52.789 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:05.897 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:02.922 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:16.030 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:02.927 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:16.035 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:02.957 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:16.065 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:02.989 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:16.098 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:02.993 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:16.061 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:02.994 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:16.103 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:03.003 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:16.072 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:03.021 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:16.129 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:03.030 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:16.099 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.830 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:25.938 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.835 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:25.944 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.878 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:25.986 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.886 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:25.955 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.888 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:25.957 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.898 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:26.007 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.901 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:26.010 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.917 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:25.986 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:12.928 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:26.036 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:17.966 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:31.074 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:17.970 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:31.078 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:17.982 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:31.090 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:18.016 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:31.085 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:18.017 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:31.126 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:18.018 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:31.086 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:18.018 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:31.127 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:18.029 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:31.098 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:18.031 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:31.140 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:32.992 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:46.100 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:33.042 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:46.110 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:33.043 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:46.112 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:33.050 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:46.158 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:33.052 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:46.161 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:33.054 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:46.162 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:33.068 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:24:46.136 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:33.079 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:24:46.188 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:33.084 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:24:46.192 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.030 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:01.138 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.034 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:01.143 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.081 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:01.189 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.100 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:01.168 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.104 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:01.173 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.110 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:01.219 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.112 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:01.220 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10088 ms) 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13077 ms) 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9911 ms) 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (16023 ms) 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9240 ms) 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (10109 ms) 2023-12-11T20:40:48.136 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: ok, hit_set contains 317:602f83fe:::foo:head 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9570 ms) 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,0 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: first is 1702327152 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,0 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,0 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,0 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,0 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,0 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,1702327157,1702327158,0 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,1702327157,1702327158,0 2023-12-11T20:40:48.137 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,1702327157,1702327158,0 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,1702327157,1702327158,1702327160,1702327161,0 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,1702327157,1702327158,1702327160,1702327161,0 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327152,1702327154,1702327155,1702327157,1702327158,1702327160,1702327161,0 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: got ls 1702327155,1702327157,1702327158,1702327160,1702327161,1702327164,1702327165,0 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: first now 1702327155, trimmed 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22119 ms) 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: foo0 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (15845 ms) 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20794 ms) 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-12-11T20:40:48.138 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24418 ms) 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5903 ms) 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4512 ms) 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4496 ms) 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3447 ms) 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (264415 ms total) 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: 2023-12-11T20:40:48.139 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-12-11T20:40:48.140 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1372139 ms total) 2023-12-11T20:40:48.140 INFO:tasks.workunit.client.0.smithi086.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-12-11T20:40:48.140 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:01.208 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92269 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92269 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92432 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92432 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92987 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92987 2023-12-11T20:40:48.141 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92734 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92734 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92573 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92573 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92352 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92352 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92684 2023-12-11T20:40:48.142 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92684 2023-12-11T20:40:48.143 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:40:48.143 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93100 2023-12-11T20:40:48.143 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93100 2023-12-11T20:40:48.143 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:01.251 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:52.915 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:06.023 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:52.965 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:06.074 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:52.977 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:06.086 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:53.011 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:06.119 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:53.016 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:06.085 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:53.017 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:06.125 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:53.018 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:06.087 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:53.028 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:06.137 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:40:53.029 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:06.097 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:07.995 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:21.102 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:07.999 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:21.107 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:08.014 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:21.123 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:08.046 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:21.154 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:08.049 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:21.158 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:08.053 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:21.121 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:08.054 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:21.123 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:08.064 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:21.173 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:08.067 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:21.136 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.009 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:26.117 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.014 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:26.122 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.057 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:26.165 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.062 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:26.131 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.067 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:26.135 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.074 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:26.183 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.077 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:26.185 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.077 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:26.146 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:13.087 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:26.195 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.117 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:41.225 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.119 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:41.228 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.163 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:41.232 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.165 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:41.234 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.167 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:41.276 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.168 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:41.276 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.172 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:41.280 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.205 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:41.274 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:28.222 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:41.330 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.088 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:46.196 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.094 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:46.203 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.108 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:25:46.216 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.132 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:46.200 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.135 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:46.203 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.139 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:46.247 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.142 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:46.250 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.147 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:25:46.216 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:33.155 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:25:46.264 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.120 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:01.228 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.123 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:01.232 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.136 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:01.244 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.171 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:01.279 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.171 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:01.240 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.172 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:01.280 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.173 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:01.241 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.186 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:01.294 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:48.186 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:01.254 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.144 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:06.252 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.145 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:06.254 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.155 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:06.263 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.189 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:06.257 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.192 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:06.261 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.203 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:06.271 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.205 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:06.313 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.209 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:06.318 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:41:53.220 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:06.328 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.257 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:18.365 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.259 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:18.367 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.275 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:18.383 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.306 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:18.414 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.308 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:18.376 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.309 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:18.417 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.309 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:18.378 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.325 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:18.394 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:05.327 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:18.435 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.205 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:21.313 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.206 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:21.274 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.209 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:21.318 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.211 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:21.279 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.219 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:21.287 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.223 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:21.331 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.263 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:21.369 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.263 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:21.371 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:08.277 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:21.385 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.295 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:26.403 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.298 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:26.406 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.307 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:26.416 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.344 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:26.412 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.345 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:26.453 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.345 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:26.414 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.346 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:26.454 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.355 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:26.423 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:13.355 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:26.464 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.256 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:41.364 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.260 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:41.368 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.273 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:41.381 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.305 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:41.413 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.310 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:41.378 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.311 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:41.420 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.313 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:41.382 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.328 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:41.396 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:28.331 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:41.439 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.310 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:46.418 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.313 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:46.421 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.325 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:26:46.434 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.353 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:46.422 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.356 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:46.424 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.357 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:46.465 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.363 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:46.471 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.366 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:26:46.435 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:33.376 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:26:46.485 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.346 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:01.454 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.349 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:01.457 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.369 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:01.477 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.394 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:01.463 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.396 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:01.465 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.403 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:01.511 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.405 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:01.513 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.410 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:01.478 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:48.423 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:01.531 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.357 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:06.465 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.360 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:06.468 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.377 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:06.485 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.404 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:06.473 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.407 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:06.475 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.417 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:06.486 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.425 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:06.533 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.427 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:06.535 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:42:53.437 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:06.546 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.406 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:21.513 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.407 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:21.516 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.424 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:21.532 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.454 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:21.563 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.459 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:21.567 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.459 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:21.528 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.465 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:21.533 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.475 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:21.583 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:08.480 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:21.549 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.417 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:26.525 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.420 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:26.529 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.434 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:26.542 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.464 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:26.573 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.466 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:26.535 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.469 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:26.578 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.470 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:26.538 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.479 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:26.588 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:13.480 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:26.549 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.454 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:41.562 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.458 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:41.566 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.473 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:41.582 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.510 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:41.619 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.513 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:41.581 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.515 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:41.623 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.526 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:41.595 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.529 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:41.638 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:28.542 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:41.611 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.469 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:46.577 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.470 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:46.538 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.471 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:46.579 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.473 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:46.541 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.485 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:27:46.553 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.485 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:27:46.593 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.519 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:46.627 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.520 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:46.628 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:33.534 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:27:46.642 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.506 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:01.614 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.509 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:01.618 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.526 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:01.634 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.561 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:01.669 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.563 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:01.631 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.564 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:01.633 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.565 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:01.674 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.580 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:01.688 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:48.580 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:01.648 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.517 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:06.625 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.520 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:06.628 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.533 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:06.642 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.567 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:06.675 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.568 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:06.636 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.570 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:06.639 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.571 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:06.679 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.581 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:06.689 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:43:53.582 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:06.650 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.568 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:21.676 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.572 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:21.680 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.590 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:21.698 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.621 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:21.690 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.622 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:21.731 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.625 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:21.693 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.625 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:21.734 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.640 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:21.748 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:08.641 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:21.710 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.588 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:26.696 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.590 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:26.698 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.599 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:26.708 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.635 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:26.743 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.636 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:26.705 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.638 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:26.707 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.639 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:26.747 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.648 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:26.716 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:13.649 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:26.758 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.625 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:41.732 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.628 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:41.736 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.645 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:41.753 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.678 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:41.786 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.680 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:41.748 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.681 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:41.789 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.681 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:41.749 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.695 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:41.804 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:28.697 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:41.766 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.633 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:46.741 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.637 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:46.745 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.654 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:28:46.762 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.683 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:46.751 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.685 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:46.754 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.691 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:46.799 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.695 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:46.803 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.698 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:28:46.766 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:33.705 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:28:46.813 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.675 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:01.783 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.681 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:01.789 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.700 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:01.808 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.728 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:01.796 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.728 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:01.836 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.730 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:01.797 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.733 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:01.841 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.746 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:01.814 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:48.748 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:01.856 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.694 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:06.802 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.696 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:06.804 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.708 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:06.816 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.741 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:06.849 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.741 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:06.809 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.744 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:06.812 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.745 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:06.853 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.755 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:06.824 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:44:53.756 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:06.864 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.896 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:22.004 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.900 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:22.008 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.924 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:22.032 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.958 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:22.026 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.962 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:22.031 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.973 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:22.081 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.974 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:22.082 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.984 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:22.053 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:08.999 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:22.106 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.557 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:22.665 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.558 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:22.666 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.567 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:22.675 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.583 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:22.651 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.584 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:22.652 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.591 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:22.660 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.595 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:22.703 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.596 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:22.705 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:09.605 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:22.713 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.894 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:27.002 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.896 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:27.004 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.906 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:27.014 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.940 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:27.008 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.942 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:27.011 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.945 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:27.053 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.946 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:27.054 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.952 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:27.021 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:13.956 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:27.064 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.349 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:30.457 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.356 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:30.464 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.363 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:30.431 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.365 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:30.434 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.375 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:30.484 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.376 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:30.484 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.377 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:30.486 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.381 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:30.450 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:17.393 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:30.501 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:28.939 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:42.047 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:28.942 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:42.050 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:28.955 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:42.063 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:28.982 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:42.090 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:28.985 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:42.053 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:28.986 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:42.094 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:28.988 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:42.056 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:28.997 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:42.105 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:29.000 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:42.068 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:33.949 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:47.057 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:33.950 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:47.058 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:33.961 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:29:47.067 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:33.988 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:47.056 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:33.990 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:47.058 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:33.995 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:47.103 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:33.996 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:47.104 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:34.000 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:29:47.068 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:34.005 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:29:47.113 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:48.983 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:02.090 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:48.993 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:02.101 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:49.008 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:02.116 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:49.040 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:02.108 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:49.041 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:02.149 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:49.042 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:02.110 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:49.045 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:02.153 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:49.059 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:02.167 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:49.059 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:02.127 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:53.997 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:07.105 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:53.999 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:07.107 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:54.008 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:07.116 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:54.040 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:07.148 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:54.043 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:07.151 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:54.048 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:07.117 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:54.050 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:07.118 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:54.054 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:07.161 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:45:54.060 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:07.128 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.072 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:22.179 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.074 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:22.183 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.087 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:22.195 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.125 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:22.233 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.127 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:22.235 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.127 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:22.196 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.129 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:22.197 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.141 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:22.210 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:09.142 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:22.250 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.050 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:27.158 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.057 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:27.163 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.072 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:27.180 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.100 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:27.169 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.104 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:27.172 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.107 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:27.216 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.111 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:27.219 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.116 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:27.184 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:14.123 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:27.231 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.097 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:42.205 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.097 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:42.205 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.099 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:42.207 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.112 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:42.220 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.148 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:42.256 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.164 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:42.272 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.166 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:42.234 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.170 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:42.239 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:29.184 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:42.252 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.106 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:47.214 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.112 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:47.220 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.129 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:47.237 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.153 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:47.221 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.154 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:47.222 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.166 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:47.234 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.192 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:47.300 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.193 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:47.301 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:34.211 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:47.320 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.127 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:57.234 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.130 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:57.239 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.150 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:30:57.259 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.183 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:57.251 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.185 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:57.253 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.191 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:57.299 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.196 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:57.303 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.199 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:30:57.267 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:44.212 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:30:57.320 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.150 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:07.258 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.154 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:07.263 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.169 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:07.277 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.199 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:07.266 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.201 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:07.269 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.212 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:07.280 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.216 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:07.324 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.221 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:07.329 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:46:54.232 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:07.340 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.195 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:17.303 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.198 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:17.306 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.212 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:17.320 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.246 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:17.354 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.247 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:17.356 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.249 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:17.317 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.252 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:17.320 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.259 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:17.367 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:04.263 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:17.332 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.205 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:27.313 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.207 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:27.315 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.220 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:27.328 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.253 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:27.321 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.254 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:27.322 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.266 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:27.333 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.278 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:27.386 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.280 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:27.388 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:14.295 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:27.403 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.224 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:37.332 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.228 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:37.336 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.241 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:37.349 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.272 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:37.340 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.274 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:37.382 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.274 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:37.343 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.275 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:37.383 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.287 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:37.355 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:24.289 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:37.397 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.295 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:47.403 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.299 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:47.407 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.313 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:47.421 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.340 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:47.448 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.345 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:47.453 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.345 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:47.413 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.347 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:47.415 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.357 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:47.425 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:34.358 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:47.466 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.276 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:57.384 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.328 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:57.436 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.344 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:31:57.452 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.374 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:57.482 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.377 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:57.445 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.377 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:57.486 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.380 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:57.448 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.389 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:31:57.497 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:44.392 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:31:57.460 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.347 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:07.455 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.348 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:07.415 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.348 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:07.457 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.362 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:07.470 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.391 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:07.499 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.393 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:07.502 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.394 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:07.462 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.405 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:07.513 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:47:54.406 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:07.474 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.329 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:17.436 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.373 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:17.441 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.375 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:17.443 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.376 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:17.484 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.390 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:17.458 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.397 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:17.505 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.439 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:17.547 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.440 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:17.549 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:04.453 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:17.561 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.370 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:27.477 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.417 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:27.525 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.440 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:27.548 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.462 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:27.530 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.465 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:27.533 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.465 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:27.573 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.467 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:27.575 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.486 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:27.554 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:14.491 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:27.599 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.370 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:31.478 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.372 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:31.480 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.380 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:31.448 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.382 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:31.490 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.383 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:31.451 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.384 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:31.492 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.399 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:31.508 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.400 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:31.469 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:18.409 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:31.517 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.506 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:32.614 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.516 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:32.583 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.517 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:32.625 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.562 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:32.670 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.565 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:32.633 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.582 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:32.650 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.583 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:32.691 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.590 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:32.698 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:19.600 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:32.708 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.439 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:42.547 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.490 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:42.558 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.492 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:42.560 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.496 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:42.604 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.517 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:42.585 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.532 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:42.640 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.573 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:42.681 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.577 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:42.685 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:29.612 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:42.720 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.499 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:47.567 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.499 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:47.607 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.501 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:47.569 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.502 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:47.610 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.515 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:47.583 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.529 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:47.634 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.549 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:47.656 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.553 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:47.659 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:34.569 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:47.674 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.109 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:48.177 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.110 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:48.178 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.120 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:32:48.188 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.188 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:48.296 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.189 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:48.297 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.202 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:32:48.310 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.271 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:48.379 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.272 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:48.380 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:35.297 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:32:48.405 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.651 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:02.759 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.655 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:02.763 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.659 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:02.767 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.661 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:02.769 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.711 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:02.779 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.718 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:02.786 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.740 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:02.848 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.745 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:02.853 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:49.748 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:02.817 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.669 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:07.777 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.673 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:07.781 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.686 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:07.794 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.717 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:07.785 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.718 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:07.786 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.720 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:07.828 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.721 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:07.829 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.731 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:07.799 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:48:54.734 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:07.842 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:02.020 INFO:tasks.workunit.client.0.smithi086.stdout: io: Running main() from gmock_main.cc 2023-12-11T20:49:02.020 INFO:tasks.workunit.client.0.smithi086.stdout: io: [==========] Running 16 tests from 2 test suites. 2023-12-11T20:49:02.020 INFO:tasks.workunit.client.0.smithi086.stdout: io: [----------] Global test environment set-up. 2023-12-11T20:49:02.020 INFO:tasks.workunit.client.0.smithi086.stdout: io: [----------] 14 tests from NeoRadosIo 2023-12-11T20:49:02.020 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.Limits 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.Limits (1806413 ms) 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3364 ms) 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.ReadOp (3226 ms) 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.SparseRead (3303 ms) 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3601 ms) 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3311 ms) 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2023-12-11T20:49:02.021 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3351 ms) 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2743 ms) 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3312 ms) 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.Trunc 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.Trunc (3518 ms) 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.Remove 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.Remove (3630 ms) 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3360 ms) 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2023-12-11T20:49:02.022 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.RmXattr (3370 ms) 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3463 ms) 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [----------] 14 tests from NeoRadosIo (1849965 ms total) 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [----------] 2 tests from NeoRadosECIo 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosECIo.SimpleWrite 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosECIo.SimpleWrite (7959 ms) 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ RUN ] NeoRadosECIo.ReadOp 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ OK ] NeoRadosECIo.ReadOp (7820 ms) 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [----------] 2 tests from NeoRadosECIo (15779 ms total) 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [----------] Global test environment tear-down 2023-12-11T20:49:02.023 INFO:tasks.workunit.client.0.smithi086.stdout: io: [==========] 16 tests from 2 test suites ran. (1865744 ms total) 2023-12-11T20:49:02.024 INFO:tasks.workunit.client.0.smithi086.stdout: io: [ PASSED ] 16 tests. 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92379 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92379 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92998 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92998 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92909 2023-12-11T20:49:02.029 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92909 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92241 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92241 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92304 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92304 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92948 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92948 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92232 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92232 2023-12-11T20:49:02.030 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92254 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92254 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93220 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93220 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92502 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92502 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93311 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93311 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.031 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92624 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92624 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93045 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93045 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93347 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93347 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92536 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92536 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92967 2023-12-11T20:49:02.032 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92967 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93032 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93032 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93288 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93288 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92865 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92865 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92400 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92400 2023-12-11T20:49:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93326 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93326 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92747 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92747 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92225 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92225 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92285 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92285 2023-12-11T20:49:02.034 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92711 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92711 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92325 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92325 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92815 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92815 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=92855 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 92855 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93178 2023-12-11T20:49:02.035 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93178 2023-12-11T20:49:02.036 INFO:tasks.workunit.client.0.smithi086.stderr:+ for t in "${!pids[@]}" 2023-12-11T20:49:02.036 INFO:tasks.workunit.client.0.smithi086.stderr:+ pid=93380 2023-12-11T20:49:02.036 INFO:tasks.workunit.client.0.smithi086.stderr:+ wait 93380 2023-12-11T20:49:02.036 INFO:tasks.workunit.client.0.smithi086.stderr:+ exit 0 2023-12-11T20:49:02.036 INFO:tasks.workunit.client.0.smithi086.stderr:+ cleanup 2023-12-11T20:49:02.036 INFO:tasks.workunit.client.0.smithi086.stderr:+ pkill -P 92220 2023-12-11T20:49:02.037 INFO:tasks.workunit.client.0.smithi086.stderr:+ true 2023-12-11T20:49:02.038 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-11T20:49:02.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-11T20:49:02.068 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-12-11T20:49:02.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-11T20:49:02.378 DEBUG:teuthology.parallel:result is None 2023-12-11T20:49:02.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-11T20:49:02.405 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-11T20:49:02.405 DEBUG:teuthology.orchestra.run.smithi086:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-11T20:49:02.469 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-11T20:49:02.469 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-11T20:49:02.478 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-11T20:49:02.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-11T20:49:02.609 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.611+0000 7f28769c9640 10 monclient: get_monmap_and_config 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.611+0000 7f28769c9640 10 monclient: build_initial_monmap 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.611+0000 7f28769c9640 10 monclient: monmap: 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:02.611803+0000 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:02.611803+0000 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:02.611 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:02.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 1 Processor -- start 2023-12-11T20:49:02.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 1 -- start start 2023-12-11T20:49:02.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient: init 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f28700a8a20 0x7f287013eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient(hunting): picked mon.noname-c con 0x7f28700a8a20 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28700a8e00 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f28701423b0 0x7f2870142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient(hunting): picked mon.noname-a con 0x7f28701423b0 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28700b21f0 con 0x7f28700a8e00 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28700a9cc0 con 0x7f28701423b0 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28700a9ed0 con 0x7f28700a8a20 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.612+0000 7f28769c9640 10 monclient(hunting): authenticate will time out at 3428.941784s 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f286ffff640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f28700a8a20 0x7f287013eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f2874f66640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f28701423b0 0x7f2870142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f286ffff640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f28700a8a20 0x7f287013eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:55430/0 (socket says 172.21.15.86:55430) 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f286ffff640 1 -- 172.21.15.86:0/1818190254 learned_addr learned my addr 172.21.15.86:0/1818190254 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f286ffff640 10 monclient(hunting): get_auth_request con 0x7f28700a8a20 auth_method 0 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f286ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f286ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f286ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f2874f66640 10 monclient(hunting): get_auth_request con 0x7f28701423b0 auth_method 0 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f2874f66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f2874f66640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.613+0000 7f2874f66640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.614+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:02.614 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.614+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:02.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.614+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:02.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.614+0000 7f2874f66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:02.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.614+0000 7f2874f66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:02.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.614+0000 7f2874f66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:02.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 10 monclient(hunting): handle_auth_done global_id 20995 payload 274 2023-12-11T20:49:02.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 1 -- 172.21.15.86:0/1818190254 >> v2:172.21.15.86:3300/0 conn(0x7f28701423b0 msgr2=0x7f2870142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:02.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 1 --2- 172.21.15.86:0/1818190254 >> v2:172.21.15.86:3300/0 conn(0x7f28701423b0 0x7f2870142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:02.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 1 -- 172.21.15.86:0/1818190254 >> v1:172.21.15.86:6789/0 conn(0x7f28700a8e00 legacy=0x7f287013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:02.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 10 monclient: _finish_hunting 0 2023-12-11T20:49:02.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 1 monclient: found mon.noname-c 2023-12-11T20:49:02.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:02.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:49:02.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.615+0000 7f286ffff640 1 -- 172.21.15.86:0/1818190254 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28700aac70 con 0x7f28700a8a20 2023-12-11T20:49:02.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.617+0000 7f286ffff640 1 --2- 172.21.15.86:0/1818190254 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013eeb0 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 crypto rx=0x7f287006c390 tx=0x7f286402b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22485cc770654cfc server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:02.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 1 -- 172.21.15.86:0/1818190254 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864031750 con 0x7f28700a8a20 2023-12-11T20:49:02.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:02.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: dump: 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: _finish_auth 0 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:02.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 1 -- 172.21.15.86:0/1818190254 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28640318b0 con 0x7f28700a8a20 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 1 -- 172.21.15.86:0/1818190254 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864031c40 con 0x7f28700a8a20 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f28769c9640 5 monclient: authenticate success, global_id 20995 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f28769c9640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f28769c9640 4 set_mon_vals no callback set 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f286effd640 10 monclient: dump: 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:02.621 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:02.622 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:02.622 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:02.622 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f28769c9640 10 monclient: shutdown 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f28769c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f28769c9640 1 -- 172.21.15.86:0/1818190254 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 msgr2=0x7f287013eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.621+0000 7f28769c9640 1 --2- 172.21.15.86:0/1818190254 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013eeb0 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 crypto rx=0x7f287006c390 tx=0x7f286402b7f0 comp rx=0 tx=0).stop 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 1 -- 172.21.15.86:0/1818190254 shutdown_connections 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 1 --2- 172.21.15.86:0/1818190254 >> v2:172.21.15.86:3300/0 conn(0x7f28701423b0 0x7f2870142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 1 --2- 172.21.15.86:0/1818190254 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013eeb0 unknown :-1 s=CLOSED pgs=5102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 1 -- 172.21.15.86:0/1818190254 >> 172.21.15.86:0/1818190254 conn(0x7f28700b7870 msgr2=0x7f28700b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 1 -- 172.21.15.86:0/1818190254 shutdown_connections 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 1 -- 172.21.15.86:0/1818190254 wait complete. 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 10 monclient: build_initial_monmap 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 10 monclient: monmap: 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:02.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 1 librados: starting msgr at 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.622+0000 7f28769c9640 1 librados: starting objecter 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 Processor -- start 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 -- start start 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 librados: setting wanted keys 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 librados: calling monclient init 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient: init 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient(hunting): picked mon.noname-b con 0x7f28700a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28700a8e00 0x7f28701455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient(hunting): picked mon.noname-c con 0x7f28700a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28701423b0 0x7f2870145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:02.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient(hunting): picked mon.noname-a con 0x7f28701423b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2870146040 con 0x7f28701423b0 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2870144ec0 con 0x7f28700a8a20 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28701451a0 con 0x7f28700a8e00 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.623+0000 7f28769c9640 10 monclient(hunting): authenticate will time out at 3428.953000s 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f2874f66640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28701423b0 0x7f2870145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286ffff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28700a8e00 0x7f28701455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286ffff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:55432/0 (socket says 172.21.15.86:55432) 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286ffff640 1 -- 172.21.15.86:0/3538571555 learned_addr learned my addr 172.21.15.86:0/3538571555 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286ffff640 10 monclient(hunting): get_auth_request con 0x7f28700a8a20 auth_method 0 2023-12-11T20:49:02.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286f7fe640 10 monclient(hunting): get_auth_request con 0x7f28700a8e00 auth_method 0 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f286f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f2874f66640 10 monclient(hunting): get_auth_request con 0x7f28701423b0 auth_method 0 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f2874f66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f2874f66640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.624+0000 7f2874f66640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f286ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f286f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:02.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f286f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:02.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f286f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:02.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f2874f66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:02.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f2874f66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:02.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.625+0000 7f2874f66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 10 monclient(hunting): handle_auth_done global_id 20998 payload 824 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 1 -- 172.21.15.86:0/3538571555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28700a8e00 msgr2=0x7f28701455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28700a8e00 0x7f28701455c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 1 -- 172.21.15.86:0/3538571555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28701423b0 msgr2=0x7f2870145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28701423b0 0x7f2870145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 10 monclient: _finish_hunting 0 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 1 monclient: found mon.noname-b 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.627+0000 7f286ffff640 1 -- 172.21.15.86:0/3538571555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28640094d0 con 0x7f28700a8a20 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.629+0000 7f286ffff640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013b940 secure :-1 s=READY pgs=5103 cs=0 l=1 rev1=1 crypto rx=0x7f2864000c00 tx=0x7f28640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.629+0000 7f286cff9640 10 client.?.objecter ms_handle_connect 0x7f28700a8a20 2023-12-11T20:49:02.631 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.629+0000 7f286cff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:02.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 1 -- 172.21.15.86:0/3538571555 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28640339d0 con 0x7f28700a8a20 2023-12-11T20:49:02.632 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: dump: 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: _finish_auth 0 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: _check_auth_tickets 2023-12-11T20:49:02.633 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 1 -- 172.21.15.86:0/3538571555 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2864033b30 con 0x7f28700a8a20 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 1 -- 172.21.15.86:0/3538571555 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286403c400 con 0x7f28700a8a20 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f28769c9640 5 monclient: authenticate success, global_id 20998 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f28769c9640 10 monclient: _renew_subs 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f28769c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2870143390 con 0x7f28700a8a20 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286cff9640 10 monclient: dump: 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f286e7fc640 4 set_mon_vals no callback set 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f28769c9640 10 client.20998.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f28769c9640 10 monclient: _renew_subs 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f28769c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.633+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2870143b90 con 0x7f28700a8a20 2023-12-11T20:49:02.635 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.634+0000 7f28769c9640 1 librados: init done 2023-12-11T20:49:02.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.634+0000 7f28769c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:02.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.634+0000 7f28769c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:02.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.634+0000 7f28769c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:02.636 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.634+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287004e840 con 0x7f28700a8a20 2023-12-11T20:49:02.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.876+0000 7f286cff9640 1 -- 172.21.15.86:0/3538571555 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f286403a040 con 0x7f28700a8a20 2023-12-11T20:49:02.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.876+0000 7f286cff9640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f285c06c870 0x7f285c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:02.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.876+0000 7f286f7fe640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f285c06c870 0x7f285c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:02.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.876+0000 7f286f7fe640 10 monclient: get_auth_request con 0x7f285c06c870 auth_method 0 2023-12-11T20:49:02.876 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.877+0000 7f286f7fe640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f285c06c870 0x7f285c06ed30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f287013a790 tx=0x7f2860009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:02.876 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.877+0000 7f286cff9640 10 client.20998.objecter ms_handle_connect 0x7f285c06c870 2023-12-11T20:49:02.879 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.880+0000 7f286cff9640 1 -- 172.21.15.86:0/3538571555 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f2864053e40 con 0x7f28700a8a20 2023-12-11T20:49:02.879 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.880+0000 7f286cff9640 10 client.20998.objecter ms_dispatch 0x7f2870000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:02.879 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.880+0000 7f286cff9640 3 client.20998.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:02.879 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.880+0000 7f286cff9640 3 client.20998.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:02.879 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.880+0000 7f286cff9640 20 client.20998.objecter dump_active .. 0 homeless 2023-12-11T20:49:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.889+0000 7f286cff9640 1 -- 172.21.15.86:0/3538571555 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28640766c0 con 0x7f28700a8a20 2023-12-11T20:49:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.889+0000 7f286cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:02.889+0000 7f286cff9640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:03.075 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.077+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28700b5420 con 0x7f285c06c870 2023-12-11T20:49:03.080 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.082+0000 7f286cff9640 1 -- 172.21.15.86:0/3538571555 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151647 (secure 0 0 0) 0x7f28700b5420 con 0x7f285c06c870 2023-12-11T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-11T20:49:03.085 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 10 librados: watch_flush enter 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 10 librados: watch_flush exit 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 20 client.20998.objecter shutdown clearing up homeless session... 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 10 client.20998.objecter successfully canceled tick 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f285c06c870 msgr2=0x7f285c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f285c06c870 0x7f285c06ed30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f287013a790 tx=0x7f2860009040 comp rx=0 tx=0).stop 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 10 monclient: shutdown 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 msgr2=0x7f287013b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013b940 secure :-1 s=READY pgs=5103 cs=0 l=1 rev1=1 crypto rx=0x7f2864000c00 tx=0x7f28640046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 shutdown_connections 2023-12-11T20:49:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f285c06c870 0x7f285c06ed30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28700a8e00 0x7f28701455c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28701423b0 0x7f2870145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 --2- 172.21.15.86:0/3538571555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28700a8a20 0x7f287013b940 unknown :-1 s=CLOSED pgs=5103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 >> 172.21.15.86:0/3538571555 conn(0x7f28700b7870 msgr2=0x7f2870137320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 shutdown_connections 2023-12-11T20:49:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 -- 172.21.15.86:0/3538571555 wait complete. 2023-12-11T20:49:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.084+0000 7f28769c9640 1 librados: shutdown 2023-12-11T20:49:03.094 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":2029,"stamp":"2023-12-11T20:49:01.960159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":535100,"kb_used_data":16912,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754439620,"statfs":{"total":773094113280,"available":772546170880,"internally_reserved":0,"allocated":17317888,"data_stored":5079475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":529714737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":139,"apply_latency_ms":139,"commit_latency_ns":139000000,"apply_latency_ns":139000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.588420"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1000,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.493525+0000","last_change":"2023-12-11T20:18:00.538563+0000","last_active":"2023-12-11T20:48:58.493525+0000","last_peered":"2023-12-11T20:48:58.493525+0000","last_clean":"2023-12-11T20:48:58.493525+0000","last_became_active":"2023-12-11T20:18:00.530681+0000","last_became_peered":"2023-12-11T20:18:00.530681+0000","last_unstale":"2023-12-11T20:48:58.493525+0000","last_undegraded":"2023-12-11T20:48:58.493525+0000","last_fullsized":"2023-12-11T20:48:58.493525+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:23:20.646623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1006,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.387229+0000","last_change":"2023-12-11T20:18:01.470212+0000","last_active":"2023-12-11T20:48:57.387229+0000","last_peered":"2023-12-11T20:48:57.387229+0000","last_clean":"2023-12-11T20:48:57.387229+0000","last_became_active":"2023-12-11T20:18:01.459740+0000","last_became_peered":"2023-12-11T20:18:01.459740+0000","last_unstale":"2023-12-11T20:48:57.387229+0000","last_undegraded":"2023-12-11T20:48:57.387229+0000","last_fullsized":"2023-12-11T20:48:57.387229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:33:49.955305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1007,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:56.986254+0000","last_change":"2023-12-11T20:18:11.560096+0000","last_active":"2023-12-11T20:48:56.986254+0000","last_peered":"2023-12-11T20:48:56.986254+0000","last_clean":"2023-12-11T20:48:56.986254+0000","last_became_active":"2023-12-11T20:18:11.552993+0000","last_became_peered":"2023-12-11T20:18:11.552993+0000","last_unstale":"2023-12-11T20:48:56.986254+0000","last_undegraded":"2023-12-11T20:48:56.986254+0000","last_fullsized":"2023-12-11T20:48:56.986254+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:31:04.664827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1002,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.093452+0000","last_change":"2023-12-11T20:18:04.754752+0000","last_active":"2023-12-11T20:48:57.093452+0000","last_peered":"2023-12-11T20:48:57.093452+0000","last_clean":"2023-12-11T20:48:57.093452+0000","last_became_active":"2023-12-11T20:18:04.748213+0000","last_became_peered":"2023-12-11T20:18:04.748213+0000","last_unstale":"2023-12-11T20:48:57.093452+0000","last_undegraded":"2023-12-11T20:48:57.093452+0000","last_fullsized":"2023-12-11T20:48:57.093452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:42:40.065588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1005,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.107408+0000","last_change":"2023-12-11T20:18:07.107260+0000","last_active":"2023-12-11T20:48:57.107408+0000","last_peered":"2023-12-11T20:48:57.107408+0000","last_clean":"2023-12-11T20:48:57.107408+0000","last_became_active":"2023-12-11T20:18:06.871380+0000","last_became_peered":"2023-12-11T20:18:06.871380+0000","last_unstale":"2023-12-11T20:48:57.107408+0000","last_undegraded":"2023-12-11T20:48:57.107408+0000","last_fullsized":"2023-12-11T20:48:57.107408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:09:46.999561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1008,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.391051+0000","last_change":"2023-12-11T20:18:00.913863+0000","last_active":"2023-12-11T20:48:57.391051+0000","last_peered":"2023-12-11T20:48:57.391051+0000","last_clean":"2023-12-11T20:48:57.391051+0000","last_became_active":"2023-12-11T20:18:00.891399+0000","last_became_peered":"2023-12-11T20:18:00.891399+0000","last_unstale":"2023-12-11T20:48:57.391051+0000","last_undegraded":"2023-12-11T20:48:57.391051+0000","last_fullsized":"2023-12-11T20:48:57.391051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:30:57.714593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1009,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.002616+0000","last_change":"2023-12-11T20:18:11.329207+0000","last_active":"2023-12-11T20:48:57.002616+0000","last_peered":"2023-12-11T20:48:57.002616+0000","last_clean":"2023-12-11T20:48:57.002616+0000","last_became_active":"2023-12-11T20:18:11.323743+0000","last_became_peered":"2023-12-11T20:18:11.323743+0000","last_unstale":"2023-12-11T20:48:57.002616+0000","last_undegraded":"2023-12-11T20:48:57.002616+0000","last_fullsized":"2023-12-11T20:48:57.002616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:45:34.411191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1021,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.358567+0000","last_change":"2023-12-11T20:18:03.803423+0000","last_active":"2023-12-11T20:48:57.358567+0000","last_peered":"2023-12-11T20:48:57.358567+0000","last_clean":"2023-12-11T20:48:57.358567+0000","last_became_active":"2023-12-11T20:18:03.798156+0000","last_became_peered":"2023-12-11T20:18:03.798156+0000","last_unstale":"2023-12-11T20:48:57.358567+0000","last_undegraded":"2023-12-11T20:48:57.358567+0000","last_fullsized":"2023-12-11T20:48:57.358567+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:31:17.842399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1015,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.734172+0000","last_change":"2023-12-11T20:18:06.556522+0000","last_active":"2023-12-11T20:48:58.734172+0000","last_peered":"2023-12-11T20:48:58.734172+0000","last_clean":"2023-12-11T20:48:58.734172+0000","last_became_active":"2023-12-11T20:18:06.509738+0000","last_became_peered":"2023-12-11T20:18:06.509738+0000","last_unstale":"2023-12-11T20:48:58.734172+0000","last_undegraded":"2023-12-11T20:48:58.734172+0000","last_fullsized":"2023-12-11T20:48:58.734172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T00:20:54.755785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"118'7","reported_seq":1022,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:56.992823+0000","last_change":"2023-12-11T20:18:05.708954+0000","last_active":"2023-12-11T20:48:56.992823+0000","last_peered":"2023-12-11T20:48:56.992823+0000","last_clean":"2023-12-11T20:48:56.992823+0000","last_became_active":"2023-12-11T20:18:05.684774+0000","last_became_peered":"2023-12-11T20:18:05.684774+0000","last_unstale":"2023-12-11T20:48:56.992823+0000","last_undegraded":"2023-12-11T20:48:56.992823+0000","last_fullsized":"2023-12-11T20:48:56.992823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:35:07.384658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1005,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.090750+0000","last_change":"2023-12-11T20:18:11.631063+0000","last_active":"2023-12-11T20:48:57.090750+0000","last_peered":"2023-12-11T20:48:57.090750+0000","last_clean":"2023-12-11T20:48:57.090750+0000","last_became_active":"2023-12-11T20:18:11.576567+0000","last_became_peered":"2023-12-11T20:18:11.576567+0000","last_unstale":"2023-12-11T20:48:57.090750+0000","last_undegraded":"2023-12-11T20:48:57.090750+0000","last_fullsized":"2023-12-11T20:48:57.090750+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:22:48.857424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1021,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.353929+0000","last_change":"2023-12-11T20:18:03.266276+0000","last_active":"2023-12-11T20:48:57.353929+0000","last_peered":"2023-12-11T20:48:57.353929+0000","last_clean":"2023-12-11T20:48:57.353929+0000","last_became_active":"2023-12-11T20:18:03.237738+0000","last_became_peered":"2023-12-11T20:18:03.237738+0000","last_unstale":"2023-12-11T20:48:57.353929+0000","last_undegraded":"2023-12-11T20:48:57.353929+0000","last_fullsized":"2023-12-11T20:48:57.353929+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:00:38.864099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":1118,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.380908+0000","last_change":"2023-12-11T20:16:49.468937+0000","last_active":"2023-12-11T20:48:57.380908+0000","last_peered":"2023-12-11T20:48:57.380908+0000","last_clean":"2023-12-11T20:48:57.380908+0000","last_became_active":"2023-12-11T20:16:49.435576+0000","last_became_peered":"2023-12-11T20:16:49.435576+0000","last_unstale":"2023-12-11T20:48:57.380908+0000","last_undegraded":"2023-12-11T20:48:57.380908+0000","last_fullsized":"2023-12-11T20:48:57.380908+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_clean_scrub_stamp":"2023-12-11T20:16:45.844700+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:49:09.602489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.409096+0000","last_change":"2023-12-11T20:16:58.643847+0000","last_active":"2023-12-11T20:48:57.409096+0000","last_peered":"2023-12-11T20:48:57.409096+0000","last_clean":"2023-12-11T20:48:57.409096+0000","last_became_active":"2023-12-11T20:16:58.638544+0000","last_became_peered":"2023-12-11T20:16:58.638544+0000","last_unstale":"2023-12-11T20:48:57.409096+0000","last_undegraded":"2023-12-11T20:48:57.409096+0000","last_fullsized":"2023-12-11T20:48:57.409096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:17:51.003853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1002,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.502405+0000","last_change":"2023-12-11T20:18:12.933720+0000","last_active":"2023-12-11T20:48:58.502405+0000","last_peered":"2023-12-11T20:48:58.502405+0000","last_clean":"2023-12-11T20:48:58.502405+0000","last_became_active":"2023-12-11T20:18:12.924715+0000","last_became_peered":"2023-12-11T20:18:12.924715+0000","last_unstale":"2023-12-11T20:48:58.502405+0000","last_undegraded":"2023-12-11T20:48:58.502405+0000","last_fullsized":"2023-12-11T20:48:58.502405+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:57:59.553539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1010,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.398408+0000","last_change":"2023-12-11T20:16:58.036508+0000","last_active":"2023-12-11T20:48:57.398408+0000","last_peered":"2023-12-11T20:48:57.398408+0000","last_clean":"2023-12-11T20:48:57.398408+0000","last_became_active":"2023-12-11T20:16:58.008642+0000","last_became_peered":"2023-12-11T20:16:58.008642+0000","last_unstale":"2023-12-11T20:48:57.398408+0000","last_undegraded":"2023-12-11T20:48:57.398408+0000","last_fullsized":"2023-12-11T20:48:57.398408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:46:42.050513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1015,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.742328+0000","last_change":"2023-12-11T20:18:07.144229+0000","last_active":"2023-12-11T20:48:58.742328+0000","last_peered":"2023-12-11T20:48:58.742328+0000","last_clean":"2023-12-11T20:48:58.742328+0000","last_became_active":"2023-12-11T20:18:07.135656+0000","last_became_peered":"2023-12-11T20:18:07.135656+0000","last_unstale":"2023-12-11T20:48:58.742328+0000","last_undegraded":"2023-12-11T20:48:58.742328+0000","last_fullsized":"2023-12-11T20:48:58.742328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:48:10.044525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1026,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.344703+0000","last_change":"2023-12-11T20:16:58.236956+0000","last_active":"2023-12-11T20:48:57.344703+0000","last_peered":"2023-12-11T20:48:57.344703+0000","last_clean":"2023-12-11T20:48:57.344703+0000","last_became_active":"2023-12-11T20:16:58.208987+0000","last_became_peered":"2023-12-11T20:16:58.208987+0000","last_unstale":"2023-12-11T20:48:57.344703+0000","last_undegraded":"2023-12-11T20:48:57.344703+0000","last_fullsized":"2023-12-11T20:48:57.344703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:57:37.650191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1023,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.237706+0000","last_change":"2023-12-11T20:18:01.362777+0000","last_active":"2023-12-11T20:49:00.237706+0000","last_peered":"2023-12-11T20:49:00.237706+0000","last_clean":"2023-12-11T20:49:00.237706+0000","last_became_active":"2023-12-11T20:18:01.354083+0000","last_became_peered":"2023-12-11T20:18:01.354083+0000","last_unstale":"2023-12-11T20:49:00.237706+0000","last_undegraded":"2023-12-11T20:49:00.237706+0000","last_fullsized":"2023-12-11T20:49:00.237706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:40:18.542456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.411468+0000","last_change":"2023-12-11T20:16:58.356627+0000","last_active":"2023-12-11T20:48:57.411468+0000","last_peered":"2023-12-11T20:48:57.411468+0000","last_clean":"2023-12-11T20:48:57.411468+0000","last_became_active":"2023-12-11T20:16:58.349599+0000","last_became_peered":"2023-12-11T20:16:58.349599+0000","last_unstale":"2023-12-11T20:48:57.411468+0000","last_undegraded":"2023-12-11T20:48:57.411468+0000","last_fullsized":"2023-12-11T20:48:57.411468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:45:43.507550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1007,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.382641+0000","last_change":"2023-12-11T20:18:02.135407+0000","last_active":"2023-12-11T20:48:57.382641+0000","last_peered":"2023-12-11T20:48:57.382641+0000","last_clean":"2023-12-11T20:48:57.382641+0000","last_became_active":"2023-12-11T20:18:02.117789+0000","last_became_peered":"2023-12-11T20:18:02.117789+0000","last_unstale":"2023-12-11T20:48:57.382641+0000","last_undegraded":"2023-12-11T20:48:57.382641+0000","last_fullsized":"2023-12-11T20:48:57.382641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:59:14.324843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1020,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.752807+0000","last_change":"2023-12-11T20:16:58.412827+0000","last_active":"2023-12-11T20:48:58.752807+0000","last_peered":"2023-12-11T20:48:58.752807+0000","last_clean":"2023-12-11T20:48:58.752807+0000","last_became_active":"2023-12-11T20:16:58.383357+0000","last_became_peered":"2023-12-11T20:16:58.383357+0000","last_unstale":"2023-12-11T20:48:58.752807+0000","last_undegraded":"2023-12-11T20:48:58.752807+0000","last_fullsized":"2023-12-11T20:48:58.752807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T20:59:10.927708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1014,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.765129+0000","last_change":"2023-12-11T20:18:13.718317+0000","last_active":"2023-12-11T20:48:58.765129+0000","last_peered":"2023-12-11T20:48:58.765129+0000","last_clean":"2023-12-11T20:48:58.765129+0000","last_became_active":"2023-12-11T20:18:13.700424+0000","last_became_peered":"2023-12-11T20:18:13.700424+0000","last_unstale":"2023-12-11T20:48:58.765129+0000","last_undegraded":"2023-12-11T20:48:58.765129+0000","last_fullsized":"2023-12-11T20:48:58.765129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:57:09.221399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.428322+0000","last_change":"2023-12-11T20:16:58.680128+0000","last_active":"2023-12-11T20:48:57.428322+0000","last_peered":"2023-12-11T20:48:57.428322+0000","last_clean":"2023-12-11T20:48:57.428322+0000","last_became_active":"2023-12-11T20:16:58.674889+0000","last_became_peered":"2023-12-11T20:16:58.674889+0000","last_unstale":"2023-12-11T20:48:57.428322+0000","last_undegraded":"2023-12-11T20:48:57.428322+0000","last_fullsized":"2023-12-11T20:48:57.428322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:54:50.773022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1020,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.357034+0000","last_change":"2023-12-11T20:18:03.189633+0000","last_active":"2023-12-11T20:48:57.357034+0000","last_peered":"2023-12-11T20:48:57.357034+0000","last_clean":"2023-12-11T20:48:57.357034+0000","last_became_active":"2023-12-11T20:18:03.180691+0000","last_became_peered":"2023-12-11T20:18:03.180691+0000","last_unstale":"2023-12-11T20:48:57.357034+0000","last_undegraded":"2023-12-11T20:48:57.357034+0000","last_fullsized":"2023-12-11T20:48:57.357034+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:26:16.797482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1028,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.281191+0000","last_change":"2023-12-11T20:16:57.283249+0000","last_active":"2023-12-11T20:49:00.281191+0000","last_peered":"2023-12-11T20:49:00.281191+0000","last_clean":"2023-12-11T20:49:00.281191+0000","last_became_active":"2023-12-11T20:16:57.255904+0000","last_became_peered":"2023-12-11T20:16:57.255904+0000","last_unstale":"2023-12-11T20:49:00.281191+0000","last_undegraded":"2023-12-11T20:49:00.281191+0000","last_fullsized":"2023-12-11T20:49:00.281191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:17:34.118238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1014,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.749638+0000","last_change":"2023-12-11T20:18:33.771824+0000","last_active":"2023-12-11T20:48:58.749638+0000","last_peered":"2023-12-11T20:48:58.749638+0000","last_clean":"2023-12-11T20:48:58.749638+0000","last_became_active":"2023-12-11T20:18:33.762733+0000","last_became_peered":"2023-12-11T20:18:33.762733+0000","last_unstale":"2023-12-11T20:48:58.749638+0000","last_undegraded":"2023-12-11T20:48:58.749638+0000","last_fullsized":"2023-12-11T20:48:58.749638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:53.954274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1010,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.424106+0000","last_change":"2023-12-11T20:16:58.076986+0000","last_active":"2023-12-11T20:48:57.424106+0000","last_peered":"2023-12-11T20:48:57.424106+0000","last_clean":"2023-12-11T20:48:57.424106+0000","last_became_active":"2023-12-11T20:16:58.071647+0000","last_became_peered":"2023-12-11T20:16:58.071647+0000","last_unstale":"2023-12-11T20:48:57.424106+0000","last_undegraded":"2023-12-11T20:48:57.424106+0000","last_fullsized":"2023-12-11T20:48:57.424106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:29:07.975896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1003,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.103446+0000","last_change":"2023-12-11T20:18:11.891380+0000","last_active":"2023-12-11T20:48:57.103446+0000","last_peered":"2023-12-11T20:48:57.103446+0000","last_clean":"2023-12-11T20:48:57.103446+0000","last_became_active":"2023-12-11T20:18:11.885388+0000","last_became_peered":"2023-12-11T20:18:11.885388+0000","last_unstale":"2023-12-11T20:48:57.103446+0000","last_undegraded":"2023-12-11T20:48:57.103446+0000","last_fullsized":"2023-12-11T20:48:57.103446+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:37:07.253751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1024,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.284857+0000","last_change":"2023-12-11T20:18:12.794888+0000","last_active":"2023-12-11T20:49:00.284857+0000","last_peered":"2023-12-11T20:49:00.284857+0000","last_clean":"2023-12-11T20:49:00.284857+0000","last_became_active":"2023-12-11T20:18:12.786107+0000","last_became_peered":"2023-12-11T20:18:12.786107+0000","last_unstale":"2023-12-11T20:49:00.284857+0000","last_undegraded":"2023-12-11T20:49:00.284857+0000","last_fullsized":"2023-12-11T20:49:00.284857+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:00:25.310704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1022,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.360301+0000","last_change":"2023-12-11T20:18:00.474665+0000","last_active":"2023-12-11T20:48:57.360301+0000","last_peered":"2023-12-11T20:48:57.360301+0000","last_clean":"2023-12-11T20:48:57.360301+0000","last_became_active":"2023-12-11T20:18:00.469176+0000","last_became_peered":"2023-12-11T20:18:00.469176+0000","last_unstale":"2023-12-11T20:48:57.360301+0000","last_undegraded":"2023-12-11T20:48:57.360301+0000","last_fullsized":"2023-12-11T20:48:57.360301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:24:10.853739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1007,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:56.998991+0000","last_change":"2023-12-11T20:18:05.144024+0000","last_active":"2023-12-11T20:48:56.998991+0000","last_peered":"2023-12-11T20:48:56.998991+0000","last_clean":"2023-12-11T20:48:56.998991+0000","last_became_active":"2023-12-11T20:18:05.138035+0000","last_became_peered":"2023-12-11T20:18:05.138035+0000","last_unstale":"2023-12-11T20:48:56.998991+0000","last_undegraded":"2023-12-11T20:48:56.998991+0000","last_fullsized":"2023-12-11T20:48:56.998991+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:20:37.637313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.416205+0000","last_change":"2023-12-11T20:18:10.100025+0000","last_active":"2023-12-11T20:48:57.416205+0000","last_peered":"2023-12-11T20:48:57.416205+0000","last_clean":"2023-12-11T20:48:57.416205+0000","last_became_active":"2023-12-11T20:18:10.094694+0000","last_became_peered":"2023-12-11T20:18:10.094694+0000","last_unstale":"2023-12-11T20:48:57.416205+0000","last_undegraded":"2023-12-11T20:48:57.416205+0000","last_fullsized":"2023-12-11T20:48:57.416205+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:38:01.652138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1013,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.745464+0000","last_change":"2023-12-11T20:18:07.073119+0000","last_active":"2023-12-11T20:48:58.745464+0000","last_peered":"2023-12-11T20:48:58.745464+0000","last_clean":"2023-12-11T20:48:58.745464+0000","last_became_active":"2023-12-11T20:18:07.062338+0000","last_became_peered":"2023-12-11T20:18:07.062338+0000","last_unstale":"2023-12-11T20:48:58.745464+0000","last_undegraded":"2023-12-11T20:48:58.745464+0000","last_fullsized":"2023-12-11T20:48:58.745464+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:48:41.166067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1005,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.413077+0000","last_change":"2023-12-11T20:18:19.678310+0000","last_active":"2023-12-11T20:48:57.413077+0000","last_peered":"2023-12-11T20:48:57.413077+0000","last_clean":"2023-12-11T20:48:57.413077+0000","last_became_active":"2023-12-11T20:18:19.665316+0000","last_became_peered":"2023-12-11T20:18:19.665316+0000","last_unstale":"2023-12-11T20:48:57.413077+0000","last_undegraded":"2023-12-11T20:48:57.413077+0000","last_fullsized":"2023-12-11T20:48:57.413077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:23:40.772081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1005,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.405963+0000","last_change":"2023-12-11T20:18:18.565871+0000","last_active":"2023-12-11T20:48:57.405963+0000","last_peered":"2023-12-11T20:48:57.405963+0000","last_clean":"2023-12-11T20:48:57.405963+0000","last_became_active":"2023-12-11T20:18:18.557249+0000","last_became_peered":"2023-12-11T20:18:18.557249+0000","last_unstale":"2023-12-11T20:48:57.405963+0000","last_undegraded":"2023-12-11T20:48:57.405963+0000","last_fullsized":"2023-12-11T20:48:57.405963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:35:50.576838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.402112+0000","last_change":"2023-12-11T20:18:19.693007+0000","last_active":"2023-12-11T20:48:57.402112+0000","last_peered":"2023-12-11T20:48:57.402112+0000","last_clean":"2023-12-11T20:48:57.402112+0000","last_became_active":"2023-12-11T20:18:19.685187+0000","last_became_peered":"2023-12-11T20:18:19.685187+0000","last_unstale":"2023-12-11T20:48:57.402112+0000","last_undegraded":"2023-12-11T20:48:57.402112+0000","last_fullsized":"2023-12-11T20:48:57.402112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:01:40.810139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1006,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.395902+0000","last_change":"2023-12-11T20:18:00.559982+0000","last_active":"2023-12-11T20:48:57.395902+0000","last_peered":"2023-12-11T20:48:57.395902+0000","last_clean":"2023-12-11T20:48:57.395902+0000","last_became_active":"2023-12-11T20:18:00.554682+0000","last_became_peered":"2023-12-11T20:18:00.554682+0000","last_unstale":"2023-12-11T20:48:57.395902+0000","last_undegraded":"2023-12-11T20:48:57.395902+0000","last_fullsized":"2023-12-11T20:48:57.395902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:12:05.725839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1013,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.763610+0000","last_change":"2023-12-11T20:18:13.715085+0000","last_active":"2023-12-11T20:48:58.763610+0000","last_peered":"2023-12-11T20:48:58.763610+0000","last_clean":"2023-12-11T20:48:58.763610+0000","last_became_active":"2023-12-11T20:18:13.689623+0000","last_became_peered":"2023-12-11T20:18:13.689623+0000","last_unstale":"2023-12-11T20:48:58.763610+0000","last_undegraded":"2023-12-11T20:48:58.763610+0000","last_fullsized":"2023-12-11T20:48:58.763610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:14:13.791518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.421139+0000","last_change":"2023-12-11T20:18:10.128111+0000","last_active":"2023-12-11T20:48:57.421139+0000","last_peered":"2023-12-11T20:48:57.421139+0000","last_clean":"2023-12-11T20:48:57.421139+0000","last_became_active":"2023-12-11T20:18:10.112911+0000","last_became_peered":"2023-12-11T20:18:10.112911+0000","last_unstale":"2023-12-11T20:48:57.421139+0000","last_undegraded":"2023-12-11T20:48:57.421139+0000","last_fullsized":"2023-12-11T20:48:57.421139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:55:06.836498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1000,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.509001+0000","last_change":"2023-12-11T20:18:13.209324+0000","last_active":"2023-12-11T20:48:58.509001+0000","last_peered":"2023-12-11T20:48:58.509001+0000","last_clean":"2023-12-11T20:48:58.509001+0000","last_became_active":"2023-12-11T20:18:13.203517+0000","last_became_peered":"2023-12-11T20:18:13.203517+0000","last_unstale":"2023-12-11T20:48:58.509001+0000","last_undegraded":"2023-12-11T20:48:58.509001+0000","last_fullsized":"2023-12-11T20:48:58.509001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:09:42.156893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719477123,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69564,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302276,"statfs":{"total":96636764160,"available":96565530624,"internally_reserved":0,"allocated":2052096,"data_stored":526598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":17.427,"5min":10.535,"15min":14.893000000000001},"min":{"1min":4.5739999999999998,"5min":4.5739999999999998,"15min":4.2510000000000003},"max":{"1min":112.848,"5min":112.848,"15min":221.21299999999999},"last":24.666},{"interface":"front","average":{"1min":17.669,"5min":11.170999999999999,"15min":14.507999999999999},"min":{"1min":6.2409999999999997,"5min":5.3739999999999997,"15min":4.7930000000000001},"max":{"1min":94.400999999999996,"5min":94.400999999999996,"15min":164.01900000000001},"last":8.4009999999999998}]},{"osd":1,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":21.391999999999999,"5min":12.878,"15min":16.920999999999999},"min":{"1min":7.6660000000000004,"5min":6.8419999999999996,"15min":5.0380000000000003},"max":{"1min":117.63,"5min":117.63,"15min":195.87200000000001},"last":40.615000000000002},{"interface":"front","average":{"1min":26.209,"5min":17.108000000000001,"15min":20.576000000000001},"min":{"1min":10.843999999999999,"5min":9.6649999999999991,"15min":6.0300000000000002},"max":{"1min":118.309,"5min":118.309,"15min":190.53899999999999},"last":41.194000000000003}]},{"osd":2,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":15.789999999999999,"5min":11.457000000000001,"15min":18.841999999999999},"min":{"1min":6.4089999999999998,"5min":6.4089999999999998,"15min":5.8259999999999996},"max":{"1min":67.793000000000006,"5min":67.793000000000006,"15min":282.79000000000002},"last":20.920000000000002},{"interface":"front","average":{"1min":21.603000000000002,"5min":15.961,"15min":21.614000000000001},"min":{"1min":8.1829999999999998,"5min":8.1829999999999998,"15min":8.0570000000000004},"max":{"1min":68.638999999999996,"5min":68.638999999999996,"15min":200.148},"last":25.044}]},{"osd":3,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.715,"5min":15.484999999999999,"15min":19.199999999999999},"min":{"1min":9.6660000000000004,"5min":6.5270000000000001,"15min":5.3029999999999999},"max":{"1min":113.89400000000001,"5min":113.89400000000001,"15min":206.11799999999999},"last":43.728999999999999},{"interface":"front","average":{"1min":22.518000000000001,"5min":14.561,"15min":20.105},"min":{"1min":6.7809999999999997,"5min":6.7809999999999997,"15min":6.7809999999999997},"max":{"1min":140.172,"5min":140.172,"15min":171.63},"last":21.672000000000001}]},{"osd":4,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":24.693000000000001,"5min":16.344999999999999,"15min":18.094000000000001},"min":{"1min":8.5180000000000007,"5min":7.6289999999999996,"15min":6.577},"max":{"1min":110.98399999999999,"5min":110.98399999999999,"15min":188.26599999999999},"last":39.805},{"interface":"front","average":{"1min":25.422999999999998,"5min":17.34,"15min":21.085000000000001},"min":{"1min":7.9279999999999999,"5min":7.8470000000000004,"15min":7.5300000000000002},"max":{"1min":144.084,"5min":144.084,"15min":172.57400000000001},"last":23.538}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":19.388999999999999,"5min":14.335000000000001,"15min":18.338999999999999},"min":{"1min":8.4640000000000004,"5min":7.4800000000000004,"15min":6.141},"max":{"1min":62.637,"5min":62.637,"15min":201.25399999999999},"last":44.348999999999997},{"interface":"front","average":{"1min":24.920999999999999,"5min":17.582999999999998,"15min":21.760000000000002},"min":{"1min":10.914999999999999,"5min":8.9529999999999994,"15min":8.423},"max":{"1min":92.804000000000002,"5min":92.804000000000002,"15min":200.60599999999999},"last":41.959000000000003}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.809999999999999,"5min":18.367999999999999,"15min":21.443000000000001},"min":{"1min":10.18,"5min":8.4909999999999997,"15min":7.899},"max":{"1min":76.634,"5min":76.634,"15min":207.72200000000001},"last":46.040999999999997},{"interface":"front","average":{"1min":22.077000000000002,"5min":17.238,"15min":22.446999999999999},"min":{"1min":9.6129999999999995,"5min":9.3019999999999996,"15min":9.3019999999999996},"max":{"1min":69.144000000000005,"5min":69.144000000000005,"15min":193.43899999999999},"last":38.210999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477123,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65128,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306712,"statfs":{"total":96636764160,"available":96570073088,"internally_reserved":0,"allocated":2060288,"data_stored":534065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":21.152000000000001,"5min":11.818,"15min":15.324999999999999},"min":{"1min":6.3159999999999998,"5min":4.4720000000000004,"15min":4.1600000000000001},"max":{"1min":102.907,"5min":102.907,"15min":266.75400000000002},"last":11.272},{"interface":"front","average":{"1min":22.907,"5min":12.157999999999999,"15min":16.690999999999999},"min":{"1min":5.4100000000000001,"5min":5.4100000000000001,"15min":4.5330000000000004},"max":{"1min":110.068,"5min":110.068,"15min":228.12899999999999},"last":9.0730000000000004}]},{"osd":1,"last update":"Mon Dec 11 20:48:39 2023","interfaces":[{"interface":"back","average":{"1min":42.024000000000001,"5min":17.748999999999999,"15min":16.431000000000001},"min":{"1min":7.1529999999999996,"5min":6.7489999999999997,"15min":5.6180000000000003},"max":{"1min":170.31200000000001,"5min":170.31200000000001,"15min":179.11000000000001},"last":10.391},{"interface":"front","average":{"1min":50.826000000000001,"5min":19.158999999999999,"15min":18.561},"min":{"1min":8.9849999999999994,"5min":7.0529999999999999,"15min":6.508},"max":{"1min":215.578,"5min":215.578,"15min":215.578},"last":13.814}]},{"osd":2,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":27.824000000000002,"5min":15.458,"15min":18.983000000000001},"min":{"1min":8.6820000000000004,"5min":8.1479999999999997,"15min":7.1150000000000002},"max":{"1min":146.12700000000001,"5min":146.12700000000001,"15min":226.56299999999999},"last":9.423},{"interface":"front","average":{"1min":30.817,"5min":17.829000000000001,"15min":21.931000000000001},"min":{"1min":11.138999999999999,"5min":10.387,"15min":9.5990000000000002},"max":{"1min":156.46199999999999,"5min":156.46199999999999,"15min":227.40899999999999},"last":19.469999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:12 2023","interfaces":[{"interface":"back","average":{"1min":26.216999999999999,"5min":17.390000000000001,"15min":20.870000000000001},"min":{"1min":10.401999999999999,"5min":10.08,"15min":9.6539999999999999},"max":{"1min":91.260000000000005,"5min":91.260000000000005,"15min":246.178},"last":16.042000000000002},{"interface":"front","average":{"1min":21.318999999999999,"5min":13.746,"15min":16.047999999999998},"min":{"1min":7.742,"5min":7.0510000000000002,"15min":7.0510000000000002},"max":{"1min":82.622,"5min":82.622,"15min":173.66800000000001},"last":11.932}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":40.828000000000003,"5min":19.824000000000002,"15min":23.663},"min":{"1min":10.592000000000001,"5min":7.5250000000000004,"15min":7.5250000000000004},"max":{"1min":119.569,"5min":119.569,"15min":246.55000000000001},"last":14.65},{"interface":"front","average":{"1min":49.973999999999997,"5min":16.795999999999999,"15min":17.120999999999999},"min":{"1min":6.2610000000000001,"5min":5.4429999999999996,"15min":5.4429999999999996},"max":{"1min":203.21799999999999,"5min":203.21799999999999,"15min":205.15799999999999},"last":6.2759999999999998}]},{"osd":6,"last update":"Mon Dec 11 20:48:12 2023","interfaces":[{"interface":"back","average":{"1min":30.149999999999999,"5min":19.039000000000001,"15min":25.846},"min":{"1min":11.76,"5min":10.831,"15min":9.9320000000000004},"max":{"1min":114.998,"5min":114.998,"15min":266.13499999999999},"last":17.385000000000002},{"interface":"front","average":{"1min":28.504000000000001,"5min":17.774999999999999,"15min":23.965},"min":{"1min":11.42,"5min":9.8339999999999996,"15min":8.391},"max":{"1min":115.377,"5min":115.377,"15min":208.00299999999999},"last":16.800000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":60.445,"5min":26.123000000000001,"15min":21.013999999999999},"min":{"1min":12.372,"5min":12.295999999999999,"15min":6.4690000000000003},"max":{"1min":260.45999999999998,"5min":260.45999999999998,"15min":260.45999999999998},"last":20.302},{"interface":"front","average":{"1min":44.287999999999997,"5min":22.015999999999998,"15min":25.881},"min":{"1min":12.273999999999999,"5min":10.863,"15min":10.863},"max":{"1min":172.559,"5min":172.559,"15min":264.99299999999999},"last":18.449999999999999}]}]},{"osd":2,"up_from":16,"seq":68719477122,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65216,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306624,"statfs":{"total":96636764160,"available":96569982976,"internally_reserved":0,"allocated":2048000,"data_stored":526593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":20.559000000000001,"5min":12.036,"15min":15.512},"min":{"1min":5.0780000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"max":{"1min":89.881,"5min":89.881,"15min":200.447},"last":18.100999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":11.49,"15min":16.253},"min":{"1min":5.6619999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":92.114000000000004,"5min":92.114000000000004,"15min":198.22900000000001},"last":19.876999999999999}]},{"osd":1,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":21.713000000000001,"5min":13.699,"15min":16.245999999999999},"min":{"1min":7.0590000000000002,"5min":5.7869999999999999,"15min":5.7869999999999999},"max":{"1min":89.183999999999997,"5min":89.183999999999997,"15min":143.56},"last":7.1269999999999998},{"interface":"front","average":{"1min":27.739999999999998,"5min":15.618,"15min":17.497},"min":{"1min":6.1879999999999997,"5min":6.1879999999999997,"15min":6.1879999999999997},"max":{"1min":175.16200000000001,"5min":175.16200000000001,"15min":175.16200000000001},"last":31.603000000000002}]},{"osd":3,"last update":"Mon Dec 11 20:48:04 2023","interfaces":[{"interface":"back","average":{"1min":8.5960000000000001,"5min":9.266,"15min":13.256},"min":{"1min":5.7960000000000003,"5min":5.3550000000000004,"15min":5.3550000000000004},"max":{"1min":15.034000000000001,"5min":21.725999999999999,"15min":171.71700000000001},"last":18.925000000000001},{"interface":"front","average":{"1min":13.776999999999999,"5min":13.532999999999999,"15min":17.481000000000002},"min":{"1min":9.9380000000000006,"5min":6.7169999999999996,"15min":5.9749999999999996},"max":{"1min":20.417000000000002,"5min":22.396000000000001,"15min":182.63399999999999},"last":24.898}]},{"osd":4,"last update":"Mon Dec 11 20:48:04 2023","interfaces":[{"interface":"back","average":{"1min":13.311999999999999,"5min":13.516,"15min":18.062999999999999},"min":{"1min":8.7010000000000005,"5min":7.6349999999999998,"15min":7.6349999999999998},"max":{"1min":18.588999999999999,"5min":27.902999999999999,"15min":161.84899999999999},"last":17.302},{"interface":"front","average":{"1min":16.510000000000002,"5min":17.042000000000002,"15min":22.538},"min":{"1min":10.923999999999999,"5min":10.923999999999999,"15min":9.9870000000000001},"max":{"1min":21.420999999999999,"5min":31.385999999999999,"15min":183.25899999999999},"last":21.986000000000001}]},{"osd":5,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":49.709000000000003,"5min":22.446999999999999,"15min":21.960999999999999},"min":{"1min":7.8159999999999998,"5min":7.8159999999999998,"15min":7.8159999999999998},"max":{"1min":364.11200000000002,"5min":364.11200000000002,"15min":364.11200000000002},"last":34.593000000000004},{"interface":"front","average":{"1min":39.561999999999998,"5min":20.593,"15min":24.591999999999999},"min":{"1min":10.625999999999999,"5min":9.1349999999999998,"15min":8.5020000000000007},"max":{"1min":296.74000000000001,"5min":296.74000000000001,"15min":296.74000000000001},"last":25.559000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:18 2023","interfaces":[{"interface":"back","average":{"1min":24.114999999999998,"5min":14.093999999999999,"15min":17.513999999999999},"min":{"1min":6.6799999999999997,"5min":6.6799999999999997,"15min":6.0990000000000002},"max":{"1min":251.19300000000001,"5min":251.19300000000001,"15min":251.19300000000001},"last":27.727},{"interface":"front","average":{"1min":17.216000000000001,"5min":16.463999999999999,"15min":21.641999999999999},"min":{"1min":12.948,"5min":9.4960000000000004,"15min":9.4469999999999992},"max":{"1min":30.542000000000002,"5min":30.542000000000002,"15min":181.96299999999999},"last":26.811}]},{"osd":7,"last update":"Mon Dec 11 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":17.181999999999999,"5min":18.201000000000001,"15min":23.617000000000001},"min":{"1min":10.379,"5min":10.379,"15min":10.379},"max":{"1min":22.295999999999999,"5min":30.533000000000001,"15min":197.27199999999999},"last":33.119},{"interface":"front","average":{"1min":13.143000000000001,"5min":13.51,"15min":18.437999999999999},"min":{"1min":8.0709999999999997,"5min":8.0199999999999996,"15min":7.085},"max":{"1min":18.725000000000001,"5min":23.817,"15min":233.57400000000001},"last":28.289000000000001}]}]},{"osd":1,"up_from":16,"seq":68719477123,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69784,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302056,"statfs":{"total":96636764160,"available":96565305344,"internally_reserved":0,"allocated":2461696,"data_stored":931905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":7.8470000000000004,"15min":13.646000000000001},"min":{"1min":4.6870000000000003,"5min":4.0949999999999998,"15min":3.903},"max":{"1min":72.888000000000005,"5min":72.888000000000005,"15min":283.91899999999998},"last":6.0129999999999999},{"interface":"front","average":{"1min":17.266999999999999,"5min":9.9909999999999997,"15min":14.855},"min":{"1min":5.8250000000000002,"5min":4.9340000000000002,"15min":4.6879999999999997},"max":{"1min":99.680000000000007,"5min":99.680000000000007,"15min":391.19999999999999},"last":20.667999999999999}]},{"osd":2,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":18.952999999999999,"5min":13.276,"15min":19.867999999999999},"min":{"1min":7.7510000000000003,"5min":6.0300000000000002,"15min":6.0300000000000002},"max":{"1min":84.167000000000002,"5min":84.167000000000002,"15min":389.66800000000001},"last":8.6199999999999992},{"interface":"front","average":{"1min":19.573,"5min":14.933999999999999,"15min":20.850000000000001},"min":{"1min":9.2989999999999995,"5min":9.2590000000000003,"15min":6.7670000000000003},"max":{"1min":100.836,"5min":100.836,"15min":376.82100000000003},"last":9.7769999999999992}]},{"osd":3,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":20.777999999999999,"5min":12.682,"15min":24.100999999999999},"min":{"1min":6.9080000000000004,"5min":5.7610000000000001,"15min":5.7610000000000001},"max":{"1min":101.74299999999999,"5min":101.74299999999999,"15min":421.89400000000001},"last":24.378},{"interface":"front","average":{"1min":29.260000000000002,"5min":16.670000000000002,"15min":20.762},"min":{"1min":10.404,"5min":5.7220000000000004,"15min":5.7220000000000004},"max":{"1min":175.047,"5min":175.047,"15min":293.06099999999998},"last":26.821000000000002}]},{"osd":4,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":12.462999999999999,"15min":18.878},"min":{"1min":6.6740000000000004,"5min":6.1630000000000003,"15min":5.952},"max":{"1min":120.863,"5min":120.863,"15min":342.28199999999998},"last":25.181999999999999},{"interface":"front","average":{"1min":33.405000000000001,"5min":18.756,"15min":23.010000000000002},"min":{"1min":12.538,"5min":10.499000000000001,"15min":6.7039999999999997},"max":{"1min":187.15100000000001,"5min":187.15100000000001,"15min":289.68799999999999},"last":28.654}]},{"osd":5,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":25.463999999999999,"5min":17.777000000000001,"15min":22.192},"min":{"1min":11.183,"5min":11.183,"15min":7.4859999999999998},"max":{"1min":183.84700000000001,"5min":183.84700000000001,"15min":556.32500000000005},"last":29.917000000000002},{"interface":"front","average":{"1min":27.771000000000001,"5min":19.635999999999999,"15min":24.861000000000001},"min":{"1min":12.422000000000001,"5min":12.422000000000001,"15min":9.5739999999999998},"max":{"1min":186.06700000000001,"5min":186.06700000000001,"15min":309.16399999999999},"last":33.709000000000003}]},{"osd":6,"last update":"Mon Dec 11 20:48:43 2023","interfaces":[{"interface":"back","average":{"1min":23.209,"5min":13.757,"15min":20.975000000000001},"min":{"1min":8.6910000000000007,"5min":7.5270000000000001,"15min":6.7590000000000003},"max":{"1min":169.429,"5min":169.429,"15min":372.90899999999999},"last":27.663},{"interface":"front","average":{"1min":29.114999999999998,"5min":19.785,"15min":27.733000000000001},"min":{"1min":12.670999999999999,"5min":12.670999999999999,"15min":11.641999999999999},"max":{"1min":197.87299999999999,"5min":197.87299999999999,"15min":297.71499999999997},"last":35.917999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":20.015000000000001,"5min":13.327,"15min":22.792999999999999},"min":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"max":{"1min":109.782,"5min":109.782,"15min":426.10700000000003},"last":32.804000000000002},{"interface":"front","average":{"1min":33.633000000000003,"5min":21.547000000000001,"15min":29.686},"min":{"1min":13.068,"5min":13.068,"15min":11.789},"max":{"1min":189.21199999999999,"5min":189.21199999999999,"15min":331.24799999999999},"last":44.869}]}]},{"osd":0,"up_from":16,"seq":68719477125,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65416,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306424,"statfs":{"total":96636764160,"available":96569778176,"internally_reserved":0,"allocated":2064384,"data_stored":538161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":11.036,"15min":16.446000000000002},"min":{"1min":4.609,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":98.932000000000002,"5min":98.932000000000002,"15min":192.989},"last":10.585000000000001},{"interface":"front","average":{"1min":20.565000000000001,"5min":11.683999999999999,"15min":17.998999999999999},"min":{"1min":6.1719999999999997,"5min":5.2220000000000004,"15min":4.7300000000000004},"max":{"1min":101.66,"5min":101.66,"15min":288.91800000000001},"last":6.6210000000000004}]},{"osd":2,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":22.863,"5min":13.515000000000001,"15min":17.582999999999998},"min":{"1min":7.694,"5min":7.0620000000000003,"15min":5.2190000000000003},"max":{"1min":96.489999999999995,"5min":96.489999999999995,"15min":294.733},"last":5.5389999999999997},{"interface":"front","average":{"1min":29.260999999999999,"5min":17.803999999999998,"15min":23.199999999999999},"min":{"1min":11.227,"5min":9.9109999999999996,"15min":7.8630000000000004},"max":{"1min":126.831,"5min":126.831,"15min":306.08100000000002},"last":9.7769999999999992}]},{"osd":3,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.068000000000001,"5min":13.037000000000001,"15min":15.56},"min":{"1min":6.1769999999999996,"5min":5.9829999999999997,"15min":5.5979999999999999},"max":{"1min":160.999,"5min":160.999,"15min":166.49100000000001},"last":22.376000000000001},{"interface":"front","average":{"1min":29.303000000000001,"5min":17.632999999999999,"15min":20.312999999999999},"min":{"1min":10.973000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"max":{"1min":177.35400000000001,"5min":177.35400000000001,"15min":177.35400000000001},"last":11.167}]},{"osd":4,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.286000000000001,"5min":16.172999999999998,"15min":19.902999999999999},"min":{"1min":6.5970000000000004,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":169.92599999999999,"5min":169.92599999999999,"15min":169.92599999999999},"last":26.155000000000001},{"interface":"front","average":{"1min":31.260999999999999,"5min":19.547000000000001,"15min":20.800000000000001},"min":{"1min":10.371,"5min":10.371,"15min":6.6859999999999999},"max":{"1min":149.65600000000001,"5min":149.65600000000001,"15min":153.42500000000001},"last":11.763999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.992999999999999,"5min":13.356,"15min":15.753},"min":{"1min":7.0940000000000003,"5min":6.5999999999999996,"15min":6.234},"max":{"1min":110.476,"5min":110.476,"15min":210.374},"last":23.175000000000001},{"interface":"front","average":{"1min":30.498999999999999,"5min":16.635999999999999,"15min":20.227},"min":{"1min":8.7940000000000005,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":110.836,"5min":110.836,"15min":267.03199999999998},"last":28.155000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":24.010999999999999,"5min":14.885,"15min":18.350999999999999},"min":{"1min":9.0850000000000009,"5min":8.0050000000000008,"15min":6.4790000000000001},"max":{"1min":111.423,"5min":111.423,"15min":161.09899999999999},"last":101.5},{"interface":"front","average":{"1min":33.442999999999998,"5min":19.559000000000001,"15min":23.120000000000001},"min":{"1min":11.138999999999999,"5min":10.305999999999999,"15min":9.4440000000000008},"max":{"1min":186.089,"5min":186.089,"15min":246.738},"last":113.024}]},{"osd":7,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.001000000000001,"5min":19.183,"15min":21.738},"min":{"1min":9.4049999999999994,"5min":9.4049999999999994,"15min":8.8789999999999996},"max":{"1min":187.00299999999999,"5min":187.00299999999999,"15min":187.00299999999999},"last":99.403999999999996},{"interface":"front","average":{"1min":29.199999999999999,"5min":19.706,"15min":23.911999999999999},"min":{"1min":12.224,"5min":11.782999999999999,"15min":7.7640000000000002},"max":{"1min":127.563,"5min":127.563,"15min":264.45100000000002},"last":13.661}]}]},{"osd":4,"up_from":14,"seq":60129542529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65168,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306672,"statfs":{"total":96636764160,"available":96570032128,"internally_reserved":0,"allocated":2093056,"data_stored":548539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":13.959,"5min":10.147,"15min":15.048999999999999},"min":{"1min":4.5049999999999999,"5min":4.4409999999999998,"15min":4.1879999999999997},"max":{"1min":40.877000000000002,"5min":40.877000000000002,"15min":248.19900000000001},"last":6.0069999999999997},{"interface":"front","average":{"1min":17.295999999999999,"5min":10.696999999999999,"15min":13.792},"min":{"1min":5.1589999999999998,"5min":5.1059999999999999,"15min":4.1449999999999996},"max":{"1min":77.028000000000006,"5min":77.028000000000006,"15min":169.541},"last":24.709}]},{"osd":1,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":23.628,"5min":14.050000000000001,"15min":18.832000000000001},"min":{"1min":7.1269999999999998,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":116.298,"5min":116.298,"15min":310.53500000000003},"last":7.1269999999999998},{"interface":"front","average":{"1min":21.18,"5min":13.920999999999999,"15min":18.367999999999999},"min":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":4.7380000000000004},"max":{"1min":109.753,"5min":109.753,"15min":269.65100000000001},"last":25.824000000000002}]},{"osd":2,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":14.571999999999999,"5min":13.747999999999999,"15min":18.155000000000001},"min":{"1min":8.3030000000000008,"5min":8.3030000000000008,"15min":5.0830000000000002},"max":{"1min":24.594000000000001,"5min":24.594000000000001,"15min":255.31700000000001},"last":9.1799999999999997},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.42,"15min":17.867000000000001},"min":{"1min":6.6429999999999998,"5min":6.6310000000000002,"15min":5.4199999999999999},"max":{"1min":16.919,"5min":25.324999999999999,"15min":291.67899999999997},"last":16.722999999999999}]},{"osd":3,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":13.087,"5min":12.433999999999999,"15min":18.640999999999998},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.2460000000000004},"max":{"1min":23.73,"5min":23.73,"15min":260.55099999999999},"last":17.395},{"interface":"front","average":{"1min":16.361000000000001,"5min":15.35,"15min":22.067},"min":{"1min":9.2949999999999999,"5min":8.1969999999999992,"15min":7.0030000000000001},"max":{"1min":20.364999999999998,"5min":21.821000000000002,"15min":320.53899999999999},"last":20.867000000000001}]},{"osd":5,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":26.222000000000001,"5min":18.422999999999998,"15min":21.838999999999999},"min":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":7.7359999999999998},"max":{"1min":132.37799999999999,"5min":132.37799999999999,"15min":298.69499999999999},"last":23.280999999999999},{"interface":"front","average":{"1min":22.783999999999999,"5min":15.782,"15min":21.024999999999999},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":81.689999999999998,"5min":81.689999999999998,"15min":309.88200000000001},"last":46.890999999999998}]},{"osd":6,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":16.067,"5min":14.606999999999999,"15min":18.742000000000001},"min":{"1min":10.874000000000001,"5min":7.6509999999999998,"15min":5.4459999999999997},"max":{"1min":30.559000000000001,"5min":30.559000000000001,"15min":290.57600000000002},"last":26.710999999999999},{"interface":"front","average":{"1min":12.907999999999999,"5min":14.144,"15min":18.532},"min":{"1min":7.3540000000000001,"5min":7.3540000000000001,"15min":6.5389999999999997},"max":{"1min":27.225999999999999,"5min":27.225999999999999,"15min":285.52499999999998},"last":28.818000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":21.823,"5min":17.431000000000001,"15min":21.483000000000001},"min":{"1min":10.587,"5min":10.176,"15min":7.2599999999999998},"max":{"1min":69.731999999999999,"5min":69.731999999999999,"15min":297.63299999999998},"last":20.187999999999999},{"interface":"front","average":{"1min":23.029,"5min":16.969999999999999,"15min":23.030000000000001},"min":{"1min":10.496,"5min":7.3639999999999999,"15min":7.3639999999999999},"max":{"1min":98.728999999999999,"5min":98.728999999999999,"15min":364.18799999999999},"last":47.399000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575236,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65128,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306712,"statfs":{"total":96636764160,"available":96570073088,"internally_reserved":0,"allocated":2060288,"data_stored":534065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:16 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":10.074999999999999,"15min":17.242999999999999},"min":{"1min":4.3600000000000003,"5min":4.3600000000000003,"15min":3.8969999999999998},"max":{"1min":15.532999999999999,"5min":16.571000000000002,"15min":280.62200000000001},"last":10.353},{"interface":"front","average":{"1min":9.0850000000000009,"5min":9.4939999999999998,"15min":17.504000000000001},"min":{"1min":6.3550000000000004,"5min":5.5990000000000002,"15min":4.9800000000000004},"max":{"1min":12.73,"5min":24.454999999999998,"15min":294.44299999999998},"last":6.101}]},{"osd":1,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":11.183999999999999,"5min":11.531000000000001,"15min":16.939},"min":{"1min":6.4340000000000002,"5min":5.5519999999999996,"15min":5.4489999999999998},"max":{"1min":18.149999999999999,"5min":18.594000000000001,"15min":217.40100000000001},"last":8.9510000000000005},{"interface":"front","average":{"1min":12.154,"5min":12.77,"15min":18.738},"min":{"1min":8.1039999999999992,"5min":6.3730000000000002,"15min":6.3730000000000002},"max":{"1min":16.456,"5min":18.507999999999999,"15min":253.15799999999999},"last":14.775}]},{"osd":2,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":12.81,"5min":13.550000000000001,"15min":18.916},"min":{"1min":7.8259999999999996,"5min":7.7839999999999998,"15min":6.319},"max":{"1min":16.373000000000001,"5min":20.346,"15min":275.31},"last":17.143000000000001},{"interface":"front","average":{"1min":15.932,"5min":16.280000000000001,"15min":24.547999999999998},"min":{"1min":11.714,"5min":11.714,"15min":11.016},"max":{"1min":19.315999999999999,"5min":22.353000000000002,"15min":399.96100000000001},"last":17.960000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":8.4879999999999995,"5min":8.3040000000000003,"15min":13.114000000000001},"min":{"1min":5.1879999999999997,"5min":5.1879999999999997,"15min":5.1879999999999997},"max":{"1min":13.698,"5min":16.690999999999999,"15min":239.00200000000001},"last":7.0570000000000004},{"interface":"front","average":{"1min":14.753,"5min":15.654999999999999,"15min":21.408000000000001},"min":{"1min":10.894,"5min":9.6389999999999993,"15min":9.2240000000000002},"max":{"1min":17.164000000000001,"5min":21.308,"15min":263.565},"last":13.917999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":27.109999999999999,"5min":15.597,"15min":16.378},"min":{"1min":9.6489999999999991,"5min":6.516,"15min":6.1900000000000004},"max":{"1min":166.67099999999999,"5min":166.67099999999999,"15min":201.459},"last":12.923999999999999},{"interface":"front","average":{"1min":22.686,"5min":13.342000000000001,"15min":17.608000000000001},"min":{"1min":7.4580000000000002,"5min":6.2510000000000003,"15min":6.2510000000000003},"max":{"1min":124.88,"5min":124.88,"15min":203.36600000000001},"last":9.9649999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":39.771000000000001,"5min":21.721,"15min":26.289000000000001},"min":{"1min":12.252000000000001,"5min":10.476000000000001,"15min":9.7810000000000006},"max":{"1min":206.904,"5min":206.904,"15min":268.98000000000002},"last":15.65},{"interface":"front","average":{"1min":28.629000000000001,"5min":16.504999999999999,"15min":19.289999999999999},"min":{"1min":8.9410000000000007,"5min":8.2639999999999993,"15min":7.6360000000000001},"max":{"1min":127.8,"5min":127.8,"15min":283.61900000000003},"last":11.708}]},{"osd":6,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":22.581,"5min":16.634,"15min":21.099},"min":{"1min":12.592000000000001,"5min":8.8339999999999996,"15min":6.6479999999999997},"max":{"1min":115.38800000000001,"5min":115.38800000000001,"15min":282.71199999999999},"last":11.077999999999999},{"interface":"front","average":{"1min":25.696999999999999,"5min":19.786000000000001,"15min":25.102},"min":{"1min":15.474,"5min":10.999000000000001,"15min":9.7569999999999997},"max":{"1min":126.97199999999999,"5min":126.97199999999999,"15min":260.13400000000001},"last":16.382000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575234,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69696,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302144,"statfs":{"total":96636764160,"available":96565395456,"internally_reserved":0,"allocated":2478080,"data_stored":939549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":13.065,"5min":10.132999999999999,"15min":13.657999999999999},"min":{"1min":4.4960000000000004,"5min":4.4509999999999996,"15min":4.3849999999999998},"max":{"1min":52.319000000000003,"5min":52.319000000000003,"15min":121.80500000000001},"last":18.257000000000001},{"interface":"front","average":{"1min":13.058999999999999,"5min":9.7390000000000008,"15min":16.152999999999999},"min":{"1min":6.3780000000000001,"5min":4.5369999999999999,"15min":4.4349999999999996},"max":{"1min":49.723999999999997,"5min":49.723999999999997,"15min":307.66000000000003},"last":6.7350000000000003}]},{"osd":1,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":12.073,"5min":9.8230000000000004,"15min":13.728},"min":{"1min":6.1070000000000002,"5min":5.3449999999999998,"15min":4.5490000000000004},"max":{"1min":57.738999999999997,"5min":57.738999999999997,"15min":131.70500000000001},"last":118.562},{"interface":"front","average":{"1min":14.811,"5min":11.605,"15min":16.902000000000001},"min":{"1min":7.4320000000000004,"5min":6.0609999999999999,"15min":5.7169999999999996},"max":{"1min":60.664999999999999,"5min":60.664999999999999,"15min":202.07300000000001},"last":24.960000000000001}]},{"osd":2,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.559000000000001,"5min":15.26,"15min":18.596},"min":{"1min":8.6720000000000006,"5min":7.7460000000000004,"15min":6.9470000000000001},"max":{"1min":109.023,"5min":109.023,"15min":203.67699999999999},"last":23.725000000000001},{"interface":"front","average":{"1min":23.201000000000001,"5min":15.288,"15min":19.117000000000001},"min":{"1min":9.0129999999999999,"5min":7.8760000000000003,"15min":6.8719999999999999},"max":{"1min":93.819999999999993,"5min":93.819999999999993,"15min":211.67500000000001},"last":46.896000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":15.050000000000001,"15min":19.510000000000002},"min":{"1min":8.3170000000000002,"5min":8.0700000000000003,"15min":7.548},"max":{"1min":58.795000000000002,"5min":58.795000000000002,"15min":167.095},"last":22.346},{"interface":"front","average":{"1min":19.649000000000001,"5min":15.371,"15min":19.989000000000001},"min":{"1min":10.052,"5min":8.1690000000000005,"15min":7.6769999999999996},"max":{"1min":68.209999999999994,"5min":68.209999999999994,"15min":207.58799999999999},"last":10.401999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.547000000000001,"5min":16.661000000000001,"15min":19.132000000000001},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":6.8360000000000003},"max":{"1min":69.195999999999998,"5min":69.195999999999998,"15min":132.64500000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":33.280999999999999,"5min":20.861000000000001,"15min":23.035},"min":{"1min":14.875999999999999,"5min":9.9390000000000001,"15min":8.2590000000000003},"max":{"1min":199.828,"5min":199.828,"15min":199.828},"last":20.135999999999999}]},{"osd":6,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":23.556999999999999,"5min":12.887,"15min":16.77},"min":{"1min":6.3620000000000001,"5min":6.234,"15min":5.8789999999999996},"max":{"1min":199.03800000000001,"5min":199.03800000000001,"15min":199.03800000000001},"last":119.57299999999999},{"interface":"front","average":{"1min":26.257999999999999,"5min":19.350999999999999,"15min":19.757000000000001},"min":{"1min":11.548999999999999,"5min":11.548999999999999,"15min":6.1479999999999997},"max":{"1min":113.911,"5min":113.911,"15min":268.072},"last":111.997}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":22.385000000000002,"5min":19.629999999999999,"15min":22.059999999999999},"min":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":8.3239999999999998},"max":{"1min":63.686999999999998,"5min":63.686999999999998,"15min":198.304},"last":20.847000000000001},{"interface":"front","average":{"1min":19.053000000000001,"5min":15.726000000000001,"15min":20.466000000000001},"min":{"1min":10.156000000000001,"5min":8.0630000000000006,"15min":8.0630000000000006},"max":{"1min":56.078000000000003,"5min":56.078000000000003,"15min":181.84299999999999},"last":7.7069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-11T20:49:03.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-11T20:49:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.201+0000 7fd4f33a4640 10 monclient: get_monmap_and_config 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.201+0000 7fd4f33a4640 10 monclient: build_initial_monmap 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.201+0000 7fd4f33a4640 10 monclient: monmap: 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:03.201702+0000 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:03.201702+0000 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.201+0000 7fd4f33a4640 1 Processor -- start 2023-12-11T20:49:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 1 -- start start 2023-12-11T20:49:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient: init 2023-12-11T20:49:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-12-11T20:49:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec0b0b80 0x7fd4ec071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ec0b0b80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd4ec13d0c0 0x7fd4ec13d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient(hunting): picked mon.noname-a con 0x7fd4ec13d0c0 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4ec143240 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ec143de0 con 0x7fd4ec143240 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ec0b0f60 con 0x7fd4ec13d0c0 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec072540 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.202+0000 7fd4f33a4640 10 monclient(hunting): authenticate will time out at 3429.531820s 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f1140640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec0b0b80 0x7fd4ec071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f1941640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fd4ec143240 0x7fd4ec1436d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:58056/0 (socket says 172.21.15.86:58056) 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f1941640 1 -- 172.21.15.86:0/3313720135 learned_addr learned my addr 172.21.15.86:0/3313720135 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f093f640 1 --2- 172.21.15.86:0/3313720135 >> v2:172.21.15.86:3300/0 conn(0x7fd4ec13d0c0 0x7fd4ec13d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f1140640 10 monclient(hunting): get_auth_request con 0x7fd4ec0b0b80 auth_method 0 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f1140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f1140640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f1140640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f093f640 10 monclient(hunting): get_auth_request con 0x7fd4ec13d0c0 auth_method 0 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f093f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.203+0000 7fd4f093f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.204+0000 7fd4f1140640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.204+0000 7fd4f1140640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.204+0000 7fd4f1140640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.204+0000 7fd4f093f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.204+0000 7fd4f093f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.204+0000 7fd4f093f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4e3fff640 1 -- 172.21.15.86:0/3313720135 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385641366 0 0) 0x7fd4ec143de0 con 0x7fd4ec143240 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4e3fff640 10 monclient(hunting): _negotiate 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4e3fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4e3fff640 10 monclient(hunting): my global_id is 21001 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4e3fff640 1 -- 172.21.15.86:0/3313720135 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d8003520 con 0x7fd4ec143240 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 10 monclient(hunting): handle_auth_done global_id 18980 payload 274 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 1 -- 172.21.15.86:0/3313720135 >> v2:172.21.15.86:3300/0 conn(0x7fd4ec13d0c0 msgr2=0x7fd4ec13d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 1 --2- 172.21.15.86:0/3313720135 >> v2:172.21.15.86:3300/0 conn(0x7fd4ec13d0c0 0x7fd4ec13d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 1 -- 172.21.15.86:0/3313720135 >> v1:172.21.15.133:6789/0 conn(0x7fd4ec143240 legacy=0x7fd4ec1436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 10 monclient: _finish_hunting 0 2023-12-11T20:49:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 1 monclient: found mon.noname-b 2023-12-11T20:49:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.205+0000 7fd4f1140640 1 -- 172.21.15.86:0/3313720135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec0a8b70 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.207+0000 7fd4f1140640 1 --2- 172.21.15.86:0/3313720135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec0b0b80 0x7fd4ec071990 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80097a0 tx=0x7fd4e802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c22205cc86d0aa3c server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 1 -- 172.21.15.86:0/3313720135 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8031830 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: dump: 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: _finish_auth 0 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: _check_auth_tickets 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 1 -- 172.21.15.86:0/3313720135 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e8031990 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4f33a4640 5 monclient: authenticate success, global_id 18980 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4f33a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4f33a4640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 1 -- 172.21.15.86:0/3313720135 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8031d20 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4f33a4640 4 set_mon_vals no callback set 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4e3fff640 10 monclient: dump: 2023-12-11T20:49:03.212 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4f33a4640 10 monclient: shutdown 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4f33a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:03.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4f33a4640 1 -- 172.21.15.86:0/3313720135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec0b0b80 msgr2=0x7fd4ec071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.211+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/3313720135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec0b0b80 0x7fd4ec071990 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80097a0 tx=0x7fd4e802b8d0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.212+0000 7fd4f33a4640 1 -- 172.21.15.86:0/3313720135 shutdown_connections 2023-12-11T20:49:03.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.212+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/3313720135 >> v2:172.21.15.86:3300/0 conn(0x7fd4ec13d0c0 0x7fd4ec13d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.212+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/3313720135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec0b0b80 0x7fd4ec071990 unknown :-1 s=CLOSED pgs=3331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.212+0000 7fd4f33a4640 1 -- 172.21.15.86:0/3313720135 >> 172.21.15.86:0/3313720135 conn(0x7fd4ec0b3ef0 msgr2=0x7fd4ec0b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:03.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.212+0000 7fd4f33a4640 1 -- 172.21.15.86:0/3313720135 shutdown_connections 2023-12-11T20:49:03.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.212+0000 7fd4f33a4640 1 -- 172.21.15.86:0/3313720135 wait complete. 2023-12-11T20:49:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.212+0000 7fd4f33a4640 10 monclient: build_initial_monmap 2023-12-11T20:49:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.212+0000 7fd4f33a4640 10 monclient: monmap: 2023-12-11T20:49:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 librados: starting msgr at 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 librados: starting objecter 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 Processor -- start 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 -- start start 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 librados: setting wanted keys 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 librados: calling monclient init 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient: init 2023-12-11T20:49:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4ec0b0b80 0x7fd4ec0ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ec0b0b80 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4ec13d0c0 0x7fd4ec0a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient(hunting): picked mon.noname-a con 0x7fd4ec13d0c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec143240 0x7fd4ec0aa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient(hunting): picked mon.noname-c con 0x7fd4ec143240 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0ab950 con 0x7fd4ec13d0c0 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0abc30 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0a9df0 con 0x7fd4ec143240 2023-12-11T20:49:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.213+0000 7fd4f33a4640 10 monclient(hunting): authenticate will time out at 3429.542943s 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1941640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec143240 0x7fd4ec0aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f093f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4ec13d0c0 0x7fd4ec0a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1140640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4ec0b0b80 0x7fd4ec0ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1941640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec143240 0x7fd4ec0aa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39268/0 (socket says 172.21.15.86:39268) 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1941640 1 -- 172.21.15.86:0/2536519310 learned_addr learned my addr 172.21.15.86:0/2536519310 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1941640 10 monclient(hunting): get_auth_request con 0x7fd4ec143240 auth_method 0 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1941640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1941640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1941640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1140640 10 monclient(hunting): get_auth_request con 0x7fd4ec0b0b80 auth_method 0 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1140640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f1140640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f093f640 10 monclient(hunting): get_auth_request con 0x7fd4ec13d0c0 auth_method 0 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f093f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.214+0000 7fd4f093f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f1941640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f1941640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f1941640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f1140640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f1140640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f1140640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f093f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f093f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.215+0000 7fd4f093f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 10 monclient(hunting): handle_auth_done global_id 21004 payload 824 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 1 -- 172.21.15.86:0/2536519310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec143240 msgr2=0x7fd4ec0aa630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec143240 0x7fd4ec0aa630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 1 -- 172.21.15.86:0/2536519310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4ec13d0c0 msgr2=0x7fd4ec0a98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4ec13d0c0 0x7fd4ec0a98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 10 monclient: _finish_hunting 0 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 1 monclient: found mon.noname-b 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.217+0000 7fd4f1140640 1 -- 172.21.15.86:0/2536519310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8009400 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.220+0000 7fd4f1140640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4ec0b0b80 0x7fd4ec0ac3a0 secure :-1 s=READY pgs=5105 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec071940 tx=0x7fd4e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.220+0000 7fd4e1ffb640 10 client.?.objecter ms_handle_connect 0x7fd4ec0b0b80 2023-12-11T20:49:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.220+0000 7fd4e1ffb640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 1 -- 172.21.15.86:0/2536519310 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e803a6f0 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: dump: 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: _finish_auth 0 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4f33a4640 5 monclient: authenticate success, global_id 21004 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 1 -- 172.21.15.86:0/2536519310 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e803a870 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 1 -- 172.21.15.86:0/2536519310 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e803ac20 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4f33a4640 10 monclient: _renew_subs 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4f33a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ec0aab70 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e37fe640 4 set_mon_vals no callback set 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.223+0000 7fd4e1ffb640 10 monclient: dump: 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.224+0000 7fd4f33a4640 10 client.21004.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.224+0000 7fd4f33a4640 10 monclient: _renew_subs 2023-12-11T20:49:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.224+0000 7fd4f33a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.224+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ec130730 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.224+0000 7fd4f33a4640 1 librados: init done 2023-12-11T20:49:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.225+0000 7fd4f33a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.225+0000 7fd4f33a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.225+0000 7fd4f33a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.225+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ec04e840 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.245+0000 7fd4e1ffb640 1 -- 172.21.15.86:0/2536519310 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd4e804c3b0 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.245+0000 7fd4e1ffb640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4d006c820 0x7fd4d006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.246+0000 7fd4f093f640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4d006c820 0x7fd4d006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.246+0000 7fd4f093f640 10 monclient: get_auth_request con 0x7fd4d006c820 auth_method 0 2023-12-11T20:49:03.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.246+0000 7fd4f093f640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4d006c820 0x7fd4d006ece0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0ac8e0 tx=0x7fd4dc009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:03.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.246+0000 7fd4e1ffb640 10 client.21004.objecter ms_handle_connect 0x7fd4d006c820 2023-12-11T20:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.470+0000 7fd4e1ffb640 1 -- 172.21.15.86:0/2536519310 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fd4e807a530 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.471+0000 7fd4e1ffb640 10 client.21004.objecter ms_dispatch 0x7fd4ec000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:03.470 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.471+0000 7fd4e1ffb640 3 client.21004.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:03.470 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.471+0000 7fd4e1ffb640 3 client.21004.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:03.470 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.471+0000 7fd4e1ffb640 20 client.21004.objecter dump_active .. 0 homeless 2023-12-11T20:49:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.480+0000 7fd4e1ffb640 1 -- 172.21.15.86:0/2536519310 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4e8076870 con 0x7fd4ec0b0b80 2023-12-11T20:49:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.480+0000 7fd4e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:03.479 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.480+0000 7fd4e1ffb640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:03.647 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.649+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4ec0a8e60 con 0x7fd4d006c820 2023-12-11T20:49:03.653 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.654+0000 7fd4e1ffb640 1 -- 172.21.15.86:0/2536519310 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151646 (secure 0 0 0) 0x7fd4dc011a30 con 0x7fd4d006c820 2023-12-11T20:49:03.654 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:49:03.655 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 10 librados: watch_flush enter 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 10 librados: watch_flush exit 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 20 client.21004.objecter shutdown clearing up homeless session... 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 10 client.21004.objecter successfully canceled tick 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4d006c820 msgr2=0x7fd4d006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4d006c820 0x7fd4d006ece0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0ac8e0 tx=0x7fd4dc009190 comp rx=0 tx=0).stop 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 10 monclient: shutdown 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4ec0b0b80 msgr2=0x7fd4ec0ac3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4ec0b0b80 0x7fd4ec0ac3a0 secure :-1 s=READY pgs=5105 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec071940 tx=0x7fd4e80046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 shutdown_connections 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4d006c820 0x7fd4d006ece0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4ec143240 0x7fd4ec0aa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4ec13d0c0 0x7fd4ec0a98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 --2- 172.21.15.86:0/2536519310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4ec0b0b80 0x7fd4ec0ac3a0 unknown :-1 s=CLOSED pgs=5105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 >> 172.21.15.86:0/2536519310 conn(0x7fd4ec0b3ef0 msgr2=0x7fd4ec0ad580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:03.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 shutdown_connections 2023-12-11T20:49:03.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.656+0000 7fd4f33a4640 1 -- 172.21.15.86:0/2536519310 wait complete. 2023-12-11T20:49:03.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.657+0000 7fd4f33a4640 1 librados: shutdown 2023-12-11T20:49:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":2030,"stamp":"2023-12-11T20:49:03.109616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":534868,"kb_used_data":16964,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754439852,"statfs":{"total":773094113280,"available":772546408448,"internally_reserved":0,"allocated":17371136,"data_stored":5109063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":529714737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":154,"apply_latency_ms":154,"commit_latency_ns":154000000,"apply_latency_ns":154000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.630347"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1000,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.493525+0000","last_change":"2023-12-11T20:18:00.538563+0000","last_active":"2023-12-11T20:48:58.493525+0000","last_peered":"2023-12-11T20:48:58.493525+0000","last_clean":"2023-12-11T20:48:58.493525+0000","last_became_active":"2023-12-11T20:18:00.530681+0000","last_became_peered":"2023-12-11T20:18:00.530681+0000","last_unstale":"2023-12-11T20:48:58.493525+0000","last_undegraded":"2023-12-11T20:48:58.493525+0000","last_fullsized":"2023-12-11T20:48:58.493525+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:23:20.646623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1006,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.387229+0000","last_change":"2023-12-11T20:18:01.470212+0000","last_active":"2023-12-11T20:48:57.387229+0000","last_peered":"2023-12-11T20:48:57.387229+0000","last_clean":"2023-12-11T20:48:57.387229+0000","last_became_active":"2023-12-11T20:18:01.459740+0000","last_became_peered":"2023-12-11T20:18:01.459740+0000","last_unstale":"2023-12-11T20:48:57.387229+0000","last_undegraded":"2023-12-11T20:48:57.387229+0000","last_fullsized":"2023-12-11T20:48:57.387229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:33:49.955305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1011,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.877928+0000","last_change":"2023-12-11T20:18:11.560096+0000","last_active":"2023-12-11T20:49:00.877928+0000","last_peered":"2023-12-11T20:49:00.877928+0000","last_clean":"2023-12-11T20:49:00.877928+0000","last_became_active":"2023-12-11T20:18:11.552993+0000","last_became_peered":"2023-12-11T20:18:11.552993+0000","last_unstale":"2023-12-11T20:49:00.877928+0000","last_undegraded":"2023-12-11T20:49:00.877928+0000","last_fullsized":"2023-12-11T20:49:00.877928+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:31:04.664827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1002,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.093452+0000","last_change":"2023-12-11T20:18:04.754752+0000","last_active":"2023-12-11T20:48:57.093452+0000","last_peered":"2023-12-11T20:48:57.093452+0000","last_clean":"2023-12-11T20:48:57.093452+0000","last_became_active":"2023-12-11T20:18:04.748213+0000","last_became_peered":"2023-12-11T20:18:04.748213+0000","last_unstale":"2023-12-11T20:48:57.093452+0000","last_undegraded":"2023-12-11T20:48:57.093452+0000","last_fullsized":"2023-12-11T20:48:57.093452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:42:40.065588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1005,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.107408+0000","last_change":"2023-12-11T20:18:07.107260+0000","last_active":"2023-12-11T20:48:57.107408+0000","last_peered":"2023-12-11T20:48:57.107408+0000","last_clean":"2023-12-11T20:48:57.107408+0000","last_became_active":"2023-12-11T20:18:06.871380+0000","last_became_peered":"2023-12-11T20:18:06.871380+0000","last_unstale":"2023-12-11T20:48:57.107408+0000","last_undegraded":"2023-12-11T20:48:57.107408+0000","last_fullsized":"2023-12-11T20:48:57.107408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:09:46.999561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1008,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.391051+0000","last_change":"2023-12-11T20:18:00.913863+0000","last_active":"2023-12-11T20:48:57.391051+0000","last_peered":"2023-12-11T20:48:57.391051+0000","last_clean":"2023-12-11T20:48:57.391051+0000","last_became_active":"2023-12-11T20:18:00.891399+0000","last_became_peered":"2023-12-11T20:18:00.891399+0000","last_unstale":"2023-12-11T20:48:57.391051+0000","last_undegraded":"2023-12-11T20:48:57.391051+0000","last_fullsized":"2023-12-11T20:48:57.391051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:30:57.714593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1013,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.892004+0000","last_change":"2023-12-11T20:18:11.329207+0000","last_active":"2023-12-11T20:49:00.892004+0000","last_peered":"2023-12-11T20:49:00.892004+0000","last_clean":"2023-12-11T20:49:00.892004+0000","last_became_active":"2023-12-11T20:18:11.323743+0000","last_became_peered":"2023-12-11T20:18:11.323743+0000","last_unstale":"2023-12-11T20:49:00.892004+0000","last_undegraded":"2023-12-11T20:49:00.892004+0000","last_fullsized":"2023-12-11T20:49:00.892004+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:45:34.411191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1021,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.358567+0000","last_change":"2023-12-11T20:18:03.803423+0000","last_active":"2023-12-11T20:48:57.358567+0000","last_peered":"2023-12-11T20:48:57.358567+0000","last_clean":"2023-12-11T20:48:57.358567+0000","last_became_active":"2023-12-11T20:18:03.798156+0000","last_became_peered":"2023-12-11T20:18:03.798156+0000","last_unstale":"2023-12-11T20:48:57.358567+0000","last_undegraded":"2023-12-11T20:48:57.358567+0000","last_fullsized":"2023-12-11T20:48:57.358567+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:31:17.842399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1015,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.734172+0000","last_change":"2023-12-11T20:18:06.556522+0000","last_active":"2023-12-11T20:48:58.734172+0000","last_peered":"2023-12-11T20:48:58.734172+0000","last_clean":"2023-12-11T20:48:58.734172+0000","last_became_active":"2023-12-11T20:18:06.509738+0000","last_became_peered":"2023-12-11T20:18:06.509738+0000","last_unstale":"2023-12-11T20:48:58.734172+0000","last_undegraded":"2023-12-11T20:48:58.734172+0000","last_fullsized":"2023-12-11T20:48:58.734172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T00:20:54.755785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"118'7","reported_seq":1026,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.874651+0000","last_change":"2023-12-11T20:18:05.708954+0000","last_active":"2023-12-11T20:49:00.874651+0000","last_peered":"2023-12-11T20:49:00.874651+0000","last_clean":"2023-12-11T20:49:00.874651+0000","last_became_active":"2023-12-11T20:18:05.684774+0000","last_became_peered":"2023-12-11T20:18:05.684774+0000","last_unstale":"2023-12-11T20:49:00.874651+0000","last_undegraded":"2023-12-11T20:49:00.874651+0000","last_fullsized":"2023-12-11T20:49:00.874651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:35:07.384658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1005,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.090750+0000","last_change":"2023-12-11T20:18:11.631063+0000","last_active":"2023-12-11T20:48:57.090750+0000","last_peered":"2023-12-11T20:48:57.090750+0000","last_clean":"2023-12-11T20:48:57.090750+0000","last_became_active":"2023-12-11T20:18:11.576567+0000","last_became_peered":"2023-12-11T20:18:11.576567+0000","last_unstale":"2023-12-11T20:48:57.090750+0000","last_undegraded":"2023-12-11T20:48:57.090750+0000","last_fullsized":"2023-12-11T20:48:57.090750+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:22:48.857424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1021,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.353929+0000","last_change":"2023-12-11T20:18:03.266276+0000","last_active":"2023-12-11T20:48:57.353929+0000","last_peered":"2023-12-11T20:48:57.353929+0000","last_clean":"2023-12-11T20:48:57.353929+0000","last_became_active":"2023-12-11T20:18:03.237738+0000","last_became_peered":"2023-12-11T20:18:03.237738+0000","last_unstale":"2023-12-11T20:48:57.353929+0000","last_undegraded":"2023-12-11T20:48:57.353929+0000","last_fullsized":"2023-12-11T20:48:57.353929+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:00:38.864099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":1118,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.380908+0000","last_change":"2023-12-11T20:16:49.468937+0000","last_active":"2023-12-11T20:48:57.380908+0000","last_peered":"2023-12-11T20:48:57.380908+0000","last_clean":"2023-12-11T20:48:57.380908+0000","last_became_active":"2023-12-11T20:16:49.435576+0000","last_became_peered":"2023-12-11T20:16:49.435576+0000","last_unstale":"2023-12-11T20:48:57.380908+0000","last_undegraded":"2023-12-11T20:48:57.380908+0000","last_fullsized":"2023-12-11T20:48:57.380908+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_clean_scrub_stamp":"2023-12-11T20:16:45.844700+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:49:09.602489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.409096+0000","last_change":"2023-12-11T20:16:58.643847+0000","last_active":"2023-12-11T20:48:57.409096+0000","last_peered":"2023-12-11T20:48:57.409096+0000","last_clean":"2023-12-11T20:48:57.409096+0000","last_became_active":"2023-12-11T20:16:58.638544+0000","last_became_peered":"2023-12-11T20:16:58.638544+0000","last_unstale":"2023-12-11T20:48:57.409096+0000","last_undegraded":"2023-12-11T20:48:57.409096+0000","last_fullsized":"2023-12-11T20:48:57.409096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:17:51.003853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1002,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.502405+0000","last_change":"2023-12-11T20:18:12.933720+0000","last_active":"2023-12-11T20:48:58.502405+0000","last_peered":"2023-12-11T20:48:58.502405+0000","last_clean":"2023-12-11T20:48:58.502405+0000","last_became_active":"2023-12-11T20:18:12.924715+0000","last_became_peered":"2023-12-11T20:18:12.924715+0000","last_unstale":"2023-12-11T20:48:58.502405+0000","last_undegraded":"2023-12-11T20:48:58.502405+0000","last_fullsized":"2023-12-11T20:48:58.502405+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:57:59.553539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1010,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.398408+0000","last_change":"2023-12-11T20:16:58.036508+0000","last_active":"2023-12-11T20:48:57.398408+0000","last_peered":"2023-12-11T20:48:57.398408+0000","last_clean":"2023-12-11T20:48:57.398408+0000","last_became_active":"2023-12-11T20:16:58.008642+0000","last_became_peered":"2023-12-11T20:16:58.008642+0000","last_unstale":"2023-12-11T20:48:57.398408+0000","last_undegraded":"2023-12-11T20:48:57.398408+0000","last_fullsized":"2023-12-11T20:48:57.398408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:46:42.050513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1015,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.742328+0000","last_change":"2023-12-11T20:18:07.144229+0000","last_active":"2023-12-11T20:48:58.742328+0000","last_peered":"2023-12-11T20:48:58.742328+0000","last_clean":"2023-12-11T20:48:58.742328+0000","last_became_active":"2023-12-11T20:18:07.135656+0000","last_became_peered":"2023-12-11T20:18:07.135656+0000","last_unstale":"2023-12-11T20:48:58.742328+0000","last_undegraded":"2023-12-11T20:48:58.742328+0000","last_fullsized":"2023-12-11T20:48:58.742328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:48:10.044525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1026,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.344703+0000","last_change":"2023-12-11T20:16:58.236956+0000","last_active":"2023-12-11T20:48:57.344703+0000","last_peered":"2023-12-11T20:48:57.344703+0000","last_clean":"2023-12-11T20:48:57.344703+0000","last_became_active":"2023-12-11T20:16:58.208987+0000","last_became_peered":"2023-12-11T20:16:58.208987+0000","last_unstale":"2023-12-11T20:48:57.344703+0000","last_undegraded":"2023-12-11T20:48:57.344703+0000","last_fullsized":"2023-12-11T20:48:57.344703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:57:37.650191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1023,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.237706+0000","last_change":"2023-12-11T20:18:01.362777+0000","last_active":"2023-12-11T20:49:00.237706+0000","last_peered":"2023-12-11T20:49:00.237706+0000","last_clean":"2023-12-11T20:49:00.237706+0000","last_became_active":"2023-12-11T20:18:01.354083+0000","last_became_peered":"2023-12-11T20:18:01.354083+0000","last_unstale":"2023-12-11T20:49:00.237706+0000","last_undegraded":"2023-12-11T20:49:00.237706+0000","last_fullsized":"2023-12-11T20:49:00.237706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:40:18.542456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.411468+0000","last_change":"2023-12-11T20:16:58.356627+0000","last_active":"2023-12-11T20:48:57.411468+0000","last_peered":"2023-12-11T20:48:57.411468+0000","last_clean":"2023-12-11T20:48:57.411468+0000","last_became_active":"2023-12-11T20:16:58.349599+0000","last_became_peered":"2023-12-11T20:16:58.349599+0000","last_unstale":"2023-12-11T20:48:57.411468+0000","last_undegraded":"2023-12-11T20:48:57.411468+0000","last_fullsized":"2023-12-11T20:48:57.411468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:45:43.507550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1007,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.382641+0000","last_change":"2023-12-11T20:18:02.135407+0000","last_active":"2023-12-11T20:48:57.382641+0000","last_peered":"2023-12-11T20:48:57.382641+0000","last_clean":"2023-12-11T20:48:57.382641+0000","last_became_active":"2023-12-11T20:18:02.117789+0000","last_became_peered":"2023-12-11T20:18:02.117789+0000","last_unstale":"2023-12-11T20:48:57.382641+0000","last_undegraded":"2023-12-11T20:48:57.382641+0000","last_fullsized":"2023-12-11T20:48:57.382641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:59:14.324843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1020,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.752807+0000","last_change":"2023-12-11T20:16:58.412827+0000","last_active":"2023-12-11T20:48:58.752807+0000","last_peered":"2023-12-11T20:48:58.752807+0000","last_clean":"2023-12-11T20:48:58.752807+0000","last_became_active":"2023-12-11T20:16:58.383357+0000","last_became_peered":"2023-12-11T20:16:58.383357+0000","last_unstale":"2023-12-11T20:48:58.752807+0000","last_undegraded":"2023-12-11T20:48:58.752807+0000","last_fullsized":"2023-12-11T20:48:58.752807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T20:59:10.927708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1014,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.765129+0000","last_change":"2023-12-11T20:18:13.718317+0000","last_active":"2023-12-11T20:48:58.765129+0000","last_peered":"2023-12-11T20:48:58.765129+0000","last_clean":"2023-12-11T20:48:58.765129+0000","last_became_active":"2023-12-11T20:18:13.700424+0000","last_became_peered":"2023-12-11T20:18:13.700424+0000","last_unstale":"2023-12-11T20:48:58.765129+0000","last_undegraded":"2023-12-11T20:48:58.765129+0000","last_fullsized":"2023-12-11T20:48:58.765129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:57:09.221399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.428322+0000","last_change":"2023-12-11T20:16:58.680128+0000","last_active":"2023-12-11T20:48:57.428322+0000","last_peered":"2023-12-11T20:48:57.428322+0000","last_clean":"2023-12-11T20:48:57.428322+0000","last_became_active":"2023-12-11T20:16:58.674889+0000","last_became_peered":"2023-12-11T20:16:58.674889+0000","last_unstale":"2023-12-11T20:48:57.428322+0000","last_undegraded":"2023-12-11T20:48:57.428322+0000","last_fullsized":"2023-12-11T20:48:57.428322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:54:50.773022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1020,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.357034+0000","last_change":"2023-12-11T20:18:03.189633+0000","last_active":"2023-12-11T20:48:57.357034+0000","last_peered":"2023-12-11T20:48:57.357034+0000","last_clean":"2023-12-11T20:48:57.357034+0000","last_became_active":"2023-12-11T20:18:03.180691+0000","last_became_peered":"2023-12-11T20:18:03.180691+0000","last_unstale":"2023-12-11T20:48:57.357034+0000","last_undegraded":"2023-12-11T20:48:57.357034+0000","last_fullsized":"2023-12-11T20:48:57.357034+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:26:16.797482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1028,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.281191+0000","last_change":"2023-12-11T20:16:57.283249+0000","last_active":"2023-12-11T20:49:00.281191+0000","last_peered":"2023-12-11T20:49:00.281191+0000","last_clean":"2023-12-11T20:49:00.281191+0000","last_became_active":"2023-12-11T20:16:57.255904+0000","last_became_peered":"2023-12-11T20:16:57.255904+0000","last_unstale":"2023-12-11T20:49:00.281191+0000","last_undegraded":"2023-12-11T20:49:00.281191+0000","last_fullsized":"2023-12-11T20:49:00.281191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:17:34.118238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1014,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.749638+0000","last_change":"2023-12-11T20:18:33.771824+0000","last_active":"2023-12-11T20:48:58.749638+0000","last_peered":"2023-12-11T20:48:58.749638+0000","last_clean":"2023-12-11T20:48:58.749638+0000","last_became_active":"2023-12-11T20:18:33.762733+0000","last_became_peered":"2023-12-11T20:18:33.762733+0000","last_unstale":"2023-12-11T20:48:58.749638+0000","last_undegraded":"2023-12-11T20:48:58.749638+0000","last_fullsized":"2023-12-11T20:48:58.749638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:53.954274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1010,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.424106+0000","last_change":"2023-12-11T20:16:58.076986+0000","last_active":"2023-12-11T20:48:57.424106+0000","last_peered":"2023-12-11T20:48:57.424106+0000","last_clean":"2023-12-11T20:48:57.424106+0000","last_became_active":"2023-12-11T20:16:58.071647+0000","last_became_peered":"2023-12-11T20:16:58.071647+0000","last_unstale":"2023-12-11T20:48:57.424106+0000","last_undegraded":"2023-12-11T20:48:57.424106+0000","last_fullsized":"2023-12-11T20:48:57.424106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:29:07.975896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1003,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.103446+0000","last_change":"2023-12-11T20:18:11.891380+0000","last_active":"2023-12-11T20:48:57.103446+0000","last_peered":"2023-12-11T20:48:57.103446+0000","last_clean":"2023-12-11T20:48:57.103446+0000","last_became_active":"2023-12-11T20:18:11.885388+0000","last_became_peered":"2023-12-11T20:18:11.885388+0000","last_unstale":"2023-12-11T20:48:57.103446+0000","last_undegraded":"2023-12-11T20:48:57.103446+0000","last_fullsized":"2023-12-11T20:48:57.103446+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:37:07.253751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1024,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.284857+0000","last_change":"2023-12-11T20:18:12.794888+0000","last_active":"2023-12-11T20:49:00.284857+0000","last_peered":"2023-12-11T20:49:00.284857+0000","last_clean":"2023-12-11T20:49:00.284857+0000","last_became_active":"2023-12-11T20:18:12.786107+0000","last_became_peered":"2023-12-11T20:18:12.786107+0000","last_unstale":"2023-12-11T20:49:00.284857+0000","last_undegraded":"2023-12-11T20:49:00.284857+0000","last_fullsized":"2023-12-11T20:49:00.284857+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:00:25.310704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1022,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.360301+0000","last_change":"2023-12-11T20:18:00.474665+0000","last_active":"2023-12-11T20:48:57.360301+0000","last_peered":"2023-12-11T20:48:57.360301+0000","last_clean":"2023-12-11T20:48:57.360301+0000","last_became_active":"2023-12-11T20:18:00.469176+0000","last_became_peered":"2023-12-11T20:18:00.469176+0000","last_unstale":"2023-12-11T20:48:57.360301+0000","last_undegraded":"2023-12-11T20:48:57.360301+0000","last_fullsized":"2023-12-11T20:48:57.360301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:24:10.853739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1011,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.886618+0000","last_change":"2023-12-11T20:18:05.144024+0000","last_active":"2023-12-11T20:49:00.886618+0000","last_peered":"2023-12-11T20:49:00.886618+0000","last_clean":"2023-12-11T20:49:00.886618+0000","last_became_active":"2023-12-11T20:18:05.138035+0000","last_became_peered":"2023-12-11T20:18:05.138035+0000","last_unstale":"2023-12-11T20:49:00.886618+0000","last_undegraded":"2023-12-11T20:49:00.886618+0000","last_fullsized":"2023-12-11T20:49:00.886618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:20:37.637313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.416205+0000","last_change":"2023-12-11T20:18:10.100025+0000","last_active":"2023-12-11T20:48:57.416205+0000","last_peered":"2023-12-11T20:48:57.416205+0000","last_clean":"2023-12-11T20:48:57.416205+0000","last_became_active":"2023-12-11T20:18:10.094694+0000","last_became_peered":"2023-12-11T20:18:10.094694+0000","last_unstale":"2023-12-11T20:48:57.416205+0000","last_undegraded":"2023-12-11T20:48:57.416205+0000","last_fullsized":"2023-12-11T20:48:57.416205+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:38:01.652138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1013,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.745464+0000","last_change":"2023-12-11T20:18:07.073119+0000","last_active":"2023-12-11T20:48:58.745464+0000","last_peered":"2023-12-11T20:48:58.745464+0000","last_clean":"2023-12-11T20:48:58.745464+0000","last_became_active":"2023-12-11T20:18:07.062338+0000","last_became_peered":"2023-12-11T20:18:07.062338+0000","last_unstale":"2023-12-11T20:48:58.745464+0000","last_undegraded":"2023-12-11T20:48:58.745464+0000","last_fullsized":"2023-12-11T20:48:58.745464+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:48:41.166067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1005,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.413077+0000","last_change":"2023-12-11T20:18:19.678310+0000","last_active":"2023-12-11T20:48:57.413077+0000","last_peered":"2023-12-11T20:48:57.413077+0000","last_clean":"2023-12-11T20:48:57.413077+0000","last_became_active":"2023-12-11T20:18:19.665316+0000","last_became_peered":"2023-12-11T20:18:19.665316+0000","last_unstale":"2023-12-11T20:48:57.413077+0000","last_undegraded":"2023-12-11T20:48:57.413077+0000","last_fullsized":"2023-12-11T20:48:57.413077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:23:40.772081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1005,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.405963+0000","last_change":"2023-12-11T20:18:18.565871+0000","last_active":"2023-12-11T20:48:57.405963+0000","last_peered":"2023-12-11T20:48:57.405963+0000","last_clean":"2023-12-11T20:48:57.405963+0000","last_became_active":"2023-12-11T20:18:18.557249+0000","last_became_peered":"2023-12-11T20:18:18.557249+0000","last_unstale":"2023-12-11T20:48:57.405963+0000","last_undegraded":"2023-12-11T20:48:57.405963+0000","last_fullsized":"2023-12-11T20:48:57.405963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:35:50.576838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.402112+0000","last_change":"2023-12-11T20:18:19.693007+0000","last_active":"2023-12-11T20:48:57.402112+0000","last_peered":"2023-12-11T20:48:57.402112+0000","last_clean":"2023-12-11T20:48:57.402112+0000","last_became_active":"2023-12-11T20:18:19.685187+0000","last_became_peered":"2023-12-11T20:18:19.685187+0000","last_unstale":"2023-12-11T20:48:57.402112+0000","last_undegraded":"2023-12-11T20:48:57.402112+0000","last_fullsized":"2023-12-11T20:48:57.402112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:01:40.810139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1006,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.395902+0000","last_change":"2023-12-11T20:18:00.559982+0000","last_active":"2023-12-11T20:48:57.395902+0000","last_peered":"2023-12-11T20:48:57.395902+0000","last_clean":"2023-12-11T20:48:57.395902+0000","last_became_active":"2023-12-11T20:18:00.554682+0000","last_became_peered":"2023-12-11T20:18:00.554682+0000","last_unstale":"2023-12-11T20:48:57.395902+0000","last_undegraded":"2023-12-11T20:48:57.395902+0000","last_fullsized":"2023-12-11T20:48:57.395902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:12:05.725839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1013,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.763610+0000","last_change":"2023-12-11T20:18:13.715085+0000","last_active":"2023-12-11T20:48:58.763610+0000","last_peered":"2023-12-11T20:48:58.763610+0000","last_clean":"2023-12-11T20:48:58.763610+0000","last_became_active":"2023-12-11T20:18:13.689623+0000","last_became_peered":"2023-12-11T20:18:13.689623+0000","last_unstale":"2023-12-11T20:48:58.763610+0000","last_undegraded":"2023-12-11T20:48:58.763610+0000","last_fullsized":"2023-12-11T20:48:58.763610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:14:13.791518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.421139+0000","last_change":"2023-12-11T20:18:10.128111+0000","last_active":"2023-12-11T20:48:57.421139+0000","last_peered":"2023-12-11T20:48:57.421139+0000","last_clean":"2023-12-11T20:48:57.421139+0000","last_became_active":"2023-12-11T20:18:10.112911+0000","last_became_peered":"2023-12-11T20:18:10.112911+0000","last_unstale":"2023-12-11T20:48:57.421139+0000","last_undegraded":"2023-12-11T20:48:57.421139+0000","last_fullsized":"2023-12-11T20:48:57.421139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:55:06.836498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1000,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.509001+0000","last_change":"2023-12-11T20:18:13.209324+0000","last_active":"2023-12-11T20:48:58.509001+0000","last_peered":"2023-12-11T20:48:58.509001+0000","last_clean":"2023-12-11T20:48:58.509001+0000","last_became_active":"2023-12-11T20:18:13.203517+0000","last_became_peered":"2023-12-11T20:18:13.203517+0000","last_unstale":"2023-12-11T20:48:58.509001+0000","last_undegraded":"2023-12-11T20:48:58.509001+0000","last_fullsized":"2023-12-11T20:48:58.509001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:09:42.156893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719477124,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69332,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302508,"statfs":{"total":96636764160,"available":96565768192,"internally_reserved":0,"allocated":2105344,"data_stored":556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":17.427,"5min":10.535,"15min":14.893000000000001},"min":{"1min":4.5739999999999998,"5min":4.5739999999999998,"15min":4.2510000000000003},"max":{"1min":112.848,"5min":112.848,"15min":221.21299999999999},"last":5.3890000000000002},{"interface":"front","average":{"1min":17.669,"5min":11.170999999999999,"15min":14.507999999999999},"min":{"1min":6.2409999999999997,"5min":5.3739999999999997,"15min":4.7930000000000001},"max":{"1min":94.400999999999996,"5min":94.400999999999996,"15min":164.01900000000001},"last":6.2089999999999996}]},{"osd":1,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":21.391999999999999,"5min":12.878,"15min":16.920999999999999},"min":{"1min":7.6660000000000004,"5min":6.8419999999999996,"15min":5.0380000000000003},"max":{"1min":117.63,"5min":117.63,"15min":195.87200000000001},"last":9.2149999999999999},{"interface":"front","average":{"1min":26.209,"5min":17.108000000000001,"15min":20.576000000000001},"min":{"1min":10.843999999999999,"5min":9.6649999999999991,"15min":6.0300000000000002},"max":{"1min":118.309,"5min":118.309,"15min":190.53899999999999},"last":10.846}]},{"osd":2,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":15.789999999999999,"5min":11.457000000000001,"15min":18.841999999999999},"min":{"1min":6.4089999999999998,"5min":6.4089999999999998,"15min":5.8259999999999996},"max":{"1min":67.793000000000006,"5min":67.793000000000006,"15min":282.79000000000002},"last":11.608000000000001},{"interface":"front","average":{"1min":21.603000000000002,"5min":15.961,"15min":21.614000000000001},"min":{"1min":8.1829999999999998,"5min":8.1829999999999998,"15min":8.0570000000000004},"max":{"1min":68.638999999999996,"5min":68.638999999999996,"15min":200.148},"last":12.178000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.715,"5min":15.484999999999999,"15min":19.199999999999999},"min":{"1min":9.6660000000000004,"5min":6.5270000000000001,"15min":5.3029999999999999},"max":{"1min":113.89400000000001,"5min":113.89400000000001,"15min":206.11799999999999},"last":12.895},{"interface":"front","average":{"1min":22.518000000000001,"5min":14.561,"15min":20.105},"min":{"1min":6.7809999999999997,"5min":6.7809999999999997,"15min":6.7809999999999997},"max":{"1min":140.172,"5min":140.172,"15min":171.63},"last":14.06}]},{"osd":4,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":24.693000000000001,"5min":16.344999999999999,"15min":18.094000000000001},"min":{"1min":8.5180000000000007,"5min":7.6289999999999996,"15min":6.577},"max":{"1min":110.98399999999999,"5min":110.98399999999999,"15min":188.26599999999999},"last":15.808},{"interface":"front","average":{"1min":25.422999999999998,"5min":17.34,"15min":21.085000000000001},"min":{"1min":7.9279999999999999,"5min":7.8470000000000004,"15min":7.5300000000000002},"max":{"1min":144.084,"5min":144.084,"15min":172.57400000000001},"last":17.742999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":19.388999999999999,"5min":14.335000000000001,"15min":18.338999999999999},"min":{"1min":8.4640000000000004,"5min":7.4800000000000004,"15min":6.141},"max":{"1min":62.637,"5min":62.637,"15min":201.25399999999999},"last":14.875999999999999},{"interface":"front","average":{"1min":24.920999999999999,"5min":17.582999999999998,"15min":21.760000000000002},"min":{"1min":10.914999999999999,"5min":8.9529999999999994,"15min":8.423},"max":{"1min":92.804000000000002,"5min":92.804000000000002,"15min":200.60599999999999},"last":16.766999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.809999999999999,"5min":18.367999999999999,"15min":21.443000000000001},"min":{"1min":10.18,"5min":8.4909999999999997,"15min":7.899},"max":{"1min":76.634,"5min":76.634,"15min":207.72200000000001},"last":38.551000000000002},{"interface":"front","average":{"1min":22.077000000000002,"5min":17.238,"15min":22.446999999999999},"min":{"1min":9.6129999999999995,"5min":9.3019999999999996,"15min":9.3019999999999996},"max":{"1min":69.144000000000005,"5min":69.144000000000005,"15min":193.43899999999999},"last":25.155999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477123,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65128,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306712,"statfs":{"total":96636764160,"available":96570073088,"internally_reserved":0,"allocated":2060288,"data_stored":534065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":21.152000000000001,"5min":11.818,"15min":15.324999999999999},"min":{"1min":6.3159999999999998,"5min":4.4720000000000004,"15min":4.1600000000000001},"max":{"1min":102.907,"5min":102.907,"15min":266.75400000000002},"last":11.272},{"interface":"front","average":{"1min":22.907,"5min":12.157999999999999,"15min":16.690999999999999},"min":{"1min":5.4100000000000001,"5min":5.4100000000000001,"15min":4.5330000000000004},"max":{"1min":110.068,"5min":110.068,"15min":228.12899999999999},"last":9.0730000000000004}]},{"osd":1,"last update":"Mon Dec 11 20:48:39 2023","interfaces":[{"interface":"back","average":{"1min":42.024000000000001,"5min":17.748999999999999,"15min":16.431000000000001},"min":{"1min":7.1529999999999996,"5min":6.7489999999999997,"15min":5.6180000000000003},"max":{"1min":170.31200000000001,"5min":170.31200000000001,"15min":179.11000000000001},"last":10.391},{"interface":"front","average":{"1min":50.826000000000001,"5min":19.158999999999999,"15min":18.561},"min":{"1min":8.9849999999999994,"5min":7.0529999999999999,"15min":6.508},"max":{"1min":215.578,"5min":215.578,"15min":215.578},"last":13.814}]},{"osd":2,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":27.824000000000002,"5min":15.458,"15min":18.983000000000001},"min":{"1min":8.6820000000000004,"5min":8.1479999999999997,"15min":7.1150000000000002},"max":{"1min":146.12700000000001,"5min":146.12700000000001,"15min":226.56299999999999},"last":9.423},{"interface":"front","average":{"1min":30.817,"5min":17.829000000000001,"15min":21.931000000000001},"min":{"1min":11.138999999999999,"5min":10.387,"15min":9.5990000000000002},"max":{"1min":156.46199999999999,"5min":156.46199999999999,"15min":227.40899999999999},"last":19.469999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:12 2023","interfaces":[{"interface":"back","average":{"1min":26.216999999999999,"5min":17.390000000000001,"15min":20.870000000000001},"min":{"1min":10.401999999999999,"5min":10.08,"15min":9.6539999999999999},"max":{"1min":91.260000000000005,"5min":91.260000000000005,"15min":246.178},"last":16.042000000000002},{"interface":"front","average":{"1min":21.318999999999999,"5min":13.746,"15min":16.047999999999998},"min":{"1min":7.742,"5min":7.0510000000000002,"15min":7.0510000000000002},"max":{"1min":82.622,"5min":82.622,"15min":173.66800000000001},"last":11.932}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":40.828000000000003,"5min":19.824000000000002,"15min":23.663},"min":{"1min":10.592000000000001,"5min":7.5250000000000004,"15min":7.5250000000000004},"max":{"1min":119.569,"5min":119.569,"15min":246.55000000000001},"last":14.65},{"interface":"front","average":{"1min":49.973999999999997,"5min":16.795999999999999,"15min":17.120999999999999},"min":{"1min":6.2610000000000001,"5min":5.4429999999999996,"15min":5.4429999999999996},"max":{"1min":203.21799999999999,"5min":203.21799999999999,"15min":205.15799999999999},"last":6.2759999999999998}]},{"osd":6,"last update":"Mon Dec 11 20:48:12 2023","interfaces":[{"interface":"back","average":{"1min":30.149999999999999,"5min":19.039000000000001,"15min":25.846},"min":{"1min":11.76,"5min":10.831,"15min":9.9320000000000004},"max":{"1min":114.998,"5min":114.998,"15min":266.13499999999999},"last":17.385000000000002},{"interface":"front","average":{"1min":28.504000000000001,"5min":17.774999999999999,"15min":23.965},"min":{"1min":11.42,"5min":9.8339999999999996,"15min":8.391},"max":{"1min":115.377,"5min":115.377,"15min":208.00299999999999},"last":16.800000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":60.445,"5min":26.123000000000001,"15min":21.013999999999999},"min":{"1min":12.372,"5min":12.295999999999999,"15min":6.4690000000000003},"max":{"1min":260.45999999999998,"5min":260.45999999999998,"15min":260.45999999999998},"last":20.302},{"interface":"front","average":{"1min":44.287999999999997,"5min":22.015999999999998,"15min":25.881},"min":{"1min":12.273999999999999,"5min":10.863,"15min":10.863},"max":{"1min":172.559,"5min":172.559,"15min":264.99299999999999},"last":18.449999999999999}]}]},{"osd":2,"up_from":16,"seq":68719477122,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65216,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306624,"statfs":{"total":96636764160,"available":96569982976,"internally_reserved":0,"allocated":2048000,"data_stored":526593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":20.559000000000001,"5min":12.036,"15min":15.512},"min":{"1min":5.0780000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"max":{"1min":89.881,"5min":89.881,"15min":200.447},"last":18.100999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":11.49,"15min":16.253},"min":{"1min":5.6619999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":92.114000000000004,"5min":92.114000000000004,"15min":198.22900000000001},"last":19.876999999999999}]},{"osd":1,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":21.713000000000001,"5min":13.699,"15min":16.245999999999999},"min":{"1min":7.0590000000000002,"5min":5.7869999999999999,"15min":5.7869999999999999},"max":{"1min":89.183999999999997,"5min":89.183999999999997,"15min":143.56},"last":7.1269999999999998},{"interface":"front","average":{"1min":27.739999999999998,"5min":15.618,"15min":17.497},"min":{"1min":6.1879999999999997,"5min":6.1879999999999997,"15min":6.1879999999999997},"max":{"1min":175.16200000000001,"5min":175.16200000000001,"15min":175.16200000000001},"last":31.603000000000002}]},{"osd":3,"last update":"Mon Dec 11 20:48:04 2023","interfaces":[{"interface":"back","average":{"1min":8.5960000000000001,"5min":9.266,"15min":13.256},"min":{"1min":5.7960000000000003,"5min":5.3550000000000004,"15min":5.3550000000000004},"max":{"1min":15.034000000000001,"5min":21.725999999999999,"15min":171.71700000000001},"last":18.925000000000001},{"interface":"front","average":{"1min":13.776999999999999,"5min":13.532999999999999,"15min":17.481000000000002},"min":{"1min":9.9380000000000006,"5min":6.7169999999999996,"15min":5.9749999999999996},"max":{"1min":20.417000000000002,"5min":22.396000000000001,"15min":182.63399999999999},"last":24.898}]},{"osd":4,"last update":"Mon Dec 11 20:48:04 2023","interfaces":[{"interface":"back","average":{"1min":13.311999999999999,"5min":13.516,"15min":18.062999999999999},"min":{"1min":8.7010000000000005,"5min":7.6349999999999998,"15min":7.6349999999999998},"max":{"1min":18.588999999999999,"5min":27.902999999999999,"15min":161.84899999999999},"last":17.302},{"interface":"front","average":{"1min":16.510000000000002,"5min":17.042000000000002,"15min":22.538},"min":{"1min":10.923999999999999,"5min":10.923999999999999,"15min":9.9870000000000001},"max":{"1min":21.420999999999999,"5min":31.385999999999999,"15min":183.25899999999999},"last":21.986000000000001}]},{"osd":5,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":49.709000000000003,"5min":22.446999999999999,"15min":21.960999999999999},"min":{"1min":7.8159999999999998,"5min":7.8159999999999998,"15min":7.8159999999999998},"max":{"1min":364.11200000000002,"5min":364.11200000000002,"15min":364.11200000000002},"last":34.593000000000004},{"interface":"front","average":{"1min":39.561999999999998,"5min":20.593,"15min":24.591999999999999},"min":{"1min":10.625999999999999,"5min":9.1349999999999998,"15min":8.5020000000000007},"max":{"1min":296.74000000000001,"5min":296.74000000000001,"15min":296.74000000000001},"last":25.559000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:18 2023","interfaces":[{"interface":"back","average":{"1min":24.114999999999998,"5min":14.093999999999999,"15min":17.513999999999999},"min":{"1min":6.6799999999999997,"5min":6.6799999999999997,"15min":6.0990000000000002},"max":{"1min":251.19300000000001,"5min":251.19300000000001,"15min":251.19300000000001},"last":27.727},{"interface":"front","average":{"1min":17.216000000000001,"5min":16.463999999999999,"15min":21.641999999999999},"min":{"1min":12.948,"5min":9.4960000000000004,"15min":9.4469999999999992},"max":{"1min":30.542000000000002,"5min":30.542000000000002,"15min":181.96299999999999},"last":26.811}]},{"osd":7,"last update":"Mon Dec 11 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":17.181999999999999,"5min":18.201000000000001,"15min":23.617000000000001},"min":{"1min":10.379,"5min":10.379,"15min":10.379},"max":{"1min":22.295999999999999,"5min":30.533000000000001,"15min":197.27199999999999},"last":33.119},{"interface":"front","average":{"1min":13.143000000000001,"5min":13.51,"15min":18.437999999999999},"min":{"1min":8.0709999999999997,"5min":8.0199999999999996,"15min":7.085},"max":{"1min":18.725000000000001,"5min":23.817,"15min":233.57400000000001},"last":28.289000000000001}]}]},{"osd":1,"up_from":16,"seq":68719477123,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69784,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302056,"statfs":{"total":96636764160,"available":96565305344,"internally_reserved":0,"allocated":2461696,"data_stored":931905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":7.8470000000000004,"15min":13.646000000000001},"min":{"1min":4.6870000000000003,"5min":4.0949999999999998,"15min":3.903},"max":{"1min":72.888000000000005,"5min":72.888000000000005,"15min":283.91899999999998},"last":6.0129999999999999},{"interface":"front","average":{"1min":17.266999999999999,"5min":9.9909999999999997,"15min":14.855},"min":{"1min":5.8250000000000002,"5min":4.9340000000000002,"15min":4.6879999999999997},"max":{"1min":99.680000000000007,"5min":99.680000000000007,"15min":391.19999999999999},"last":20.667999999999999}]},{"osd":2,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":18.952999999999999,"5min":13.276,"15min":19.867999999999999},"min":{"1min":7.7510000000000003,"5min":6.0300000000000002,"15min":6.0300000000000002},"max":{"1min":84.167000000000002,"5min":84.167000000000002,"15min":389.66800000000001},"last":8.6199999999999992},{"interface":"front","average":{"1min":19.573,"5min":14.933999999999999,"15min":20.850000000000001},"min":{"1min":9.2989999999999995,"5min":9.2590000000000003,"15min":6.7670000000000003},"max":{"1min":100.836,"5min":100.836,"15min":376.82100000000003},"last":9.7769999999999992}]},{"osd":3,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":20.777999999999999,"5min":12.682,"15min":24.100999999999999},"min":{"1min":6.9080000000000004,"5min":5.7610000000000001,"15min":5.7610000000000001},"max":{"1min":101.74299999999999,"5min":101.74299999999999,"15min":421.89400000000001},"last":24.378},{"interface":"front","average":{"1min":29.260000000000002,"5min":16.670000000000002,"15min":20.762},"min":{"1min":10.404,"5min":5.7220000000000004,"15min":5.7220000000000004},"max":{"1min":175.047,"5min":175.047,"15min":293.06099999999998},"last":26.821000000000002}]},{"osd":4,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":12.462999999999999,"15min":18.878},"min":{"1min":6.6740000000000004,"5min":6.1630000000000003,"15min":5.952},"max":{"1min":120.863,"5min":120.863,"15min":342.28199999999998},"last":25.181999999999999},{"interface":"front","average":{"1min":33.405000000000001,"5min":18.756,"15min":23.010000000000002},"min":{"1min":12.538,"5min":10.499000000000001,"15min":6.7039999999999997},"max":{"1min":187.15100000000001,"5min":187.15100000000001,"15min":289.68799999999999},"last":28.654}]},{"osd":5,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":25.463999999999999,"5min":17.777000000000001,"15min":22.192},"min":{"1min":11.183,"5min":11.183,"15min":7.4859999999999998},"max":{"1min":183.84700000000001,"5min":183.84700000000001,"15min":556.32500000000005},"last":29.917000000000002},{"interface":"front","average":{"1min":27.771000000000001,"5min":19.635999999999999,"15min":24.861000000000001},"min":{"1min":12.422000000000001,"5min":12.422000000000001,"15min":9.5739999999999998},"max":{"1min":186.06700000000001,"5min":186.06700000000001,"15min":309.16399999999999},"last":33.709000000000003}]},{"osd":6,"last update":"Mon Dec 11 20:48:43 2023","interfaces":[{"interface":"back","average":{"1min":23.209,"5min":13.757,"15min":20.975000000000001},"min":{"1min":8.6910000000000007,"5min":7.5270000000000001,"15min":6.7590000000000003},"max":{"1min":169.429,"5min":169.429,"15min":372.90899999999999},"last":27.663},{"interface":"front","average":{"1min":29.114999999999998,"5min":19.785,"15min":27.733000000000001},"min":{"1min":12.670999999999999,"5min":12.670999999999999,"15min":11.641999999999999},"max":{"1min":197.87299999999999,"5min":197.87299999999999,"15min":297.71499999999997},"last":35.917999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":20.015000000000001,"5min":13.327,"15min":22.792999999999999},"min":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"max":{"1min":109.782,"5min":109.782,"15min":426.10700000000003},"last":32.804000000000002},{"interface":"front","average":{"1min":33.633000000000003,"5min":21.547000000000001,"15min":29.686},"min":{"1min":13.068,"5min":13.068,"15min":11.789},"max":{"1min":189.21199999999999,"5min":189.21199999999999,"15min":331.24799999999999},"last":44.869}]}]},{"osd":0,"up_from":16,"seq":68719477125,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65416,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306424,"statfs":{"total":96636764160,"available":96569778176,"internally_reserved":0,"allocated":2064384,"data_stored":538161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":11.036,"15min":16.446000000000002},"min":{"1min":4.609,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":98.932000000000002,"5min":98.932000000000002,"15min":192.989},"last":10.585000000000001},{"interface":"front","average":{"1min":20.565000000000001,"5min":11.683999999999999,"15min":17.998999999999999},"min":{"1min":6.1719999999999997,"5min":5.2220000000000004,"15min":4.7300000000000004},"max":{"1min":101.66,"5min":101.66,"15min":288.91800000000001},"last":6.6210000000000004}]},{"osd":2,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":22.863,"5min":13.515000000000001,"15min":17.582999999999998},"min":{"1min":7.694,"5min":7.0620000000000003,"15min":5.2190000000000003},"max":{"1min":96.489999999999995,"5min":96.489999999999995,"15min":294.733},"last":5.5389999999999997},{"interface":"front","average":{"1min":29.260999999999999,"5min":17.803999999999998,"15min":23.199999999999999},"min":{"1min":11.227,"5min":9.9109999999999996,"15min":7.8630000000000004},"max":{"1min":126.831,"5min":126.831,"15min":306.08100000000002},"last":9.7769999999999992}]},{"osd":3,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.068000000000001,"5min":13.037000000000001,"15min":15.56},"min":{"1min":6.1769999999999996,"5min":5.9829999999999997,"15min":5.5979999999999999},"max":{"1min":160.999,"5min":160.999,"15min":166.49100000000001},"last":22.376000000000001},{"interface":"front","average":{"1min":29.303000000000001,"5min":17.632999999999999,"15min":20.312999999999999},"min":{"1min":10.973000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"max":{"1min":177.35400000000001,"5min":177.35400000000001,"15min":177.35400000000001},"last":11.167}]},{"osd":4,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.286000000000001,"5min":16.172999999999998,"15min":19.902999999999999},"min":{"1min":6.5970000000000004,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":169.92599999999999,"5min":169.92599999999999,"15min":169.92599999999999},"last":26.155000000000001},{"interface":"front","average":{"1min":31.260999999999999,"5min":19.547000000000001,"15min":20.800000000000001},"min":{"1min":10.371,"5min":10.371,"15min":6.6859999999999999},"max":{"1min":149.65600000000001,"5min":149.65600000000001,"15min":153.42500000000001},"last":11.763999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.992999999999999,"5min":13.356,"15min":15.753},"min":{"1min":7.0940000000000003,"5min":6.5999999999999996,"15min":6.234},"max":{"1min":110.476,"5min":110.476,"15min":210.374},"last":23.175000000000001},{"interface":"front","average":{"1min":30.498999999999999,"5min":16.635999999999999,"15min":20.227},"min":{"1min":8.7940000000000005,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":110.836,"5min":110.836,"15min":267.03199999999998},"last":28.155000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":24.010999999999999,"5min":14.885,"15min":18.350999999999999},"min":{"1min":9.0850000000000009,"5min":8.0050000000000008,"15min":6.4790000000000001},"max":{"1min":111.423,"5min":111.423,"15min":161.09899999999999},"last":101.5},{"interface":"front","average":{"1min":33.442999999999998,"5min":19.559000000000001,"15min":23.120000000000001},"min":{"1min":11.138999999999999,"5min":10.305999999999999,"15min":9.4440000000000008},"max":{"1min":186.089,"5min":186.089,"15min":246.738},"last":113.024}]},{"osd":7,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.001000000000001,"5min":19.183,"15min":21.738},"min":{"1min":9.4049999999999994,"5min":9.4049999999999994,"15min":8.8789999999999996},"max":{"1min":187.00299999999999,"5min":187.00299999999999,"15min":187.00299999999999},"last":99.403999999999996},{"interface":"front","average":{"1min":29.199999999999999,"5min":19.706,"15min":23.911999999999999},"min":{"1min":12.224,"5min":11.782999999999999,"15min":7.7640000000000002},"max":{"1min":127.563,"5min":127.563,"15min":264.45100000000002},"last":13.661}]}]},{"osd":4,"up_from":14,"seq":60129542529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65168,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306672,"statfs":{"total":96636764160,"available":96570032128,"internally_reserved":0,"allocated":2093056,"data_stored":548539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":13.959,"5min":10.147,"15min":15.048999999999999},"min":{"1min":4.5049999999999999,"5min":4.4409999999999998,"15min":4.1879999999999997},"max":{"1min":40.877000000000002,"5min":40.877000000000002,"15min":248.19900000000001},"last":6.0069999999999997},{"interface":"front","average":{"1min":17.295999999999999,"5min":10.696999999999999,"15min":13.792},"min":{"1min":5.1589999999999998,"5min":5.1059999999999999,"15min":4.1449999999999996},"max":{"1min":77.028000000000006,"5min":77.028000000000006,"15min":169.541},"last":24.709}]},{"osd":1,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":23.628,"5min":14.050000000000001,"15min":18.832000000000001},"min":{"1min":7.1269999999999998,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":116.298,"5min":116.298,"15min":310.53500000000003},"last":7.1269999999999998},{"interface":"front","average":{"1min":21.18,"5min":13.920999999999999,"15min":18.367999999999999},"min":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":4.7380000000000004},"max":{"1min":109.753,"5min":109.753,"15min":269.65100000000001},"last":25.824000000000002}]},{"osd":2,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":14.571999999999999,"5min":13.747999999999999,"15min":18.155000000000001},"min":{"1min":8.3030000000000008,"5min":8.3030000000000008,"15min":5.0830000000000002},"max":{"1min":24.594000000000001,"5min":24.594000000000001,"15min":255.31700000000001},"last":9.1799999999999997},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.42,"15min":17.867000000000001},"min":{"1min":6.6429999999999998,"5min":6.6310000000000002,"15min":5.4199999999999999},"max":{"1min":16.919,"5min":25.324999999999999,"15min":291.67899999999997},"last":16.722999999999999}]},{"osd":3,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":13.087,"5min":12.433999999999999,"15min":18.640999999999998},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.2460000000000004},"max":{"1min":23.73,"5min":23.73,"15min":260.55099999999999},"last":17.395},{"interface":"front","average":{"1min":16.361000000000001,"5min":15.35,"15min":22.067},"min":{"1min":9.2949999999999999,"5min":8.1969999999999992,"15min":7.0030000000000001},"max":{"1min":20.364999999999998,"5min":21.821000000000002,"15min":320.53899999999999},"last":20.867000000000001}]},{"osd":5,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":26.222000000000001,"5min":18.422999999999998,"15min":21.838999999999999},"min":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":7.7359999999999998},"max":{"1min":132.37799999999999,"5min":132.37799999999999,"15min":298.69499999999999},"last":23.280999999999999},{"interface":"front","average":{"1min":22.783999999999999,"5min":15.782,"15min":21.024999999999999},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":81.689999999999998,"5min":81.689999999999998,"15min":309.88200000000001},"last":46.890999999999998}]},{"osd":6,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":16.067,"5min":14.606999999999999,"15min":18.742000000000001},"min":{"1min":10.874000000000001,"5min":7.6509999999999998,"15min":5.4459999999999997},"max":{"1min":30.559000000000001,"5min":30.559000000000001,"15min":290.57600000000002},"last":26.710999999999999},{"interface":"front","average":{"1min":12.907999999999999,"5min":14.144,"15min":18.532},"min":{"1min":7.3540000000000001,"5min":7.3540000000000001,"15min":6.5389999999999997},"max":{"1min":27.225999999999999,"5min":27.225999999999999,"15min":285.52499999999998},"last":28.818000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":21.823,"5min":17.431000000000001,"15min":21.483000000000001},"min":{"1min":10.587,"5min":10.176,"15min":7.2599999999999998},"max":{"1min":69.731999999999999,"5min":69.731999999999999,"15min":297.63299999999998},"last":20.187999999999999},{"interface":"front","average":{"1min":23.029,"5min":16.969999999999999,"15min":23.030000000000001},"min":{"1min":10.496,"5min":7.3639999999999999,"15min":7.3639999999999999},"max":{"1min":98.728999999999999,"5min":98.728999999999999,"15min":364.18799999999999},"last":47.399000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575236,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65128,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306712,"statfs":{"total":96636764160,"available":96570073088,"internally_reserved":0,"allocated":2060288,"data_stored":534065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:16 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":10.074999999999999,"15min":17.242999999999999},"min":{"1min":4.3600000000000003,"5min":4.3600000000000003,"15min":3.8969999999999998},"max":{"1min":15.532999999999999,"5min":16.571000000000002,"15min":280.62200000000001},"last":10.353},{"interface":"front","average":{"1min":9.0850000000000009,"5min":9.4939999999999998,"15min":17.504000000000001},"min":{"1min":6.3550000000000004,"5min":5.5990000000000002,"15min":4.9800000000000004},"max":{"1min":12.73,"5min":24.454999999999998,"15min":294.44299999999998},"last":6.101}]},{"osd":1,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":11.183999999999999,"5min":11.531000000000001,"15min":16.939},"min":{"1min":6.4340000000000002,"5min":5.5519999999999996,"15min":5.4489999999999998},"max":{"1min":18.149999999999999,"5min":18.594000000000001,"15min":217.40100000000001},"last":8.9510000000000005},{"interface":"front","average":{"1min":12.154,"5min":12.77,"15min":18.738},"min":{"1min":8.1039999999999992,"5min":6.3730000000000002,"15min":6.3730000000000002},"max":{"1min":16.456,"5min":18.507999999999999,"15min":253.15799999999999},"last":14.775}]},{"osd":2,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":12.81,"5min":13.550000000000001,"15min":18.916},"min":{"1min":7.8259999999999996,"5min":7.7839999999999998,"15min":6.319},"max":{"1min":16.373000000000001,"5min":20.346,"15min":275.31},"last":17.143000000000001},{"interface":"front","average":{"1min":15.932,"5min":16.280000000000001,"15min":24.547999999999998},"min":{"1min":11.714,"5min":11.714,"15min":11.016},"max":{"1min":19.315999999999999,"5min":22.353000000000002,"15min":399.96100000000001},"last":17.960000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":8.4879999999999995,"5min":8.3040000000000003,"15min":13.114000000000001},"min":{"1min":5.1879999999999997,"5min":5.1879999999999997,"15min":5.1879999999999997},"max":{"1min":13.698,"5min":16.690999999999999,"15min":239.00200000000001},"last":7.0570000000000004},{"interface":"front","average":{"1min":14.753,"5min":15.654999999999999,"15min":21.408000000000001},"min":{"1min":10.894,"5min":9.6389999999999993,"15min":9.2240000000000002},"max":{"1min":17.164000000000001,"5min":21.308,"15min":263.565},"last":13.917999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":27.109999999999999,"5min":15.597,"15min":16.378},"min":{"1min":9.6489999999999991,"5min":6.516,"15min":6.1900000000000004},"max":{"1min":166.67099999999999,"5min":166.67099999999999,"15min":201.459},"last":12.923999999999999},{"interface":"front","average":{"1min":22.686,"5min":13.342000000000001,"15min":17.608000000000001},"min":{"1min":7.4580000000000002,"5min":6.2510000000000003,"15min":6.2510000000000003},"max":{"1min":124.88,"5min":124.88,"15min":203.36600000000001},"last":9.9649999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":39.771000000000001,"5min":21.721,"15min":26.289000000000001},"min":{"1min":12.252000000000001,"5min":10.476000000000001,"15min":9.7810000000000006},"max":{"1min":206.904,"5min":206.904,"15min":268.98000000000002},"last":15.65},{"interface":"front","average":{"1min":28.629000000000001,"5min":16.504999999999999,"15min":19.289999999999999},"min":{"1min":8.9410000000000007,"5min":8.2639999999999993,"15min":7.6360000000000001},"max":{"1min":127.8,"5min":127.8,"15min":283.61900000000003},"last":11.708}]},{"osd":6,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":22.581,"5min":16.634,"15min":21.099},"min":{"1min":12.592000000000001,"5min":8.8339999999999996,"15min":6.6479999999999997},"max":{"1min":115.38800000000001,"5min":115.38800000000001,"15min":282.71199999999999},"last":11.077999999999999},{"interface":"front","average":{"1min":25.696999999999999,"5min":19.786000000000001,"15min":25.102},"min":{"1min":15.474,"5min":10.999000000000001,"15min":9.7569999999999997},"max":{"1min":126.97199999999999,"5min":126.97199999999999,"15min":260.13400000000001},"last":16.382000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575234,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69696,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302144,"statfs":{"total":96636764160,"available":96565395456,"internally_reserved":0,"allocated":2478080,"data_stored":939549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":13.065,"5min":10.132999999999999,"15min":13.657999999999999},"min":{"1min":4.4960000000000004,"5min":4.4509999999999996,"15min":4.3849999999999998},"max":{"1min":52.319000000000003,"5min":52.319000000000003,"15min":121.80500000000001},"last":18.257000000000001},{"interface":"front","average":{"1min":13.058999999999999,"5min":9.7390000000000008,"15min":16.152999999999999},"min":{"1min":6.3780000000000001,"5min":4.5369999999999999,"15min":4.4349999999999996},"max":{"1min":49.723999999999997,"5min":49.723999999999997,"15min":307.66000000000003},"last":6.7350000000000003}]},{"osd":1,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":12.073,"5min":9.8230000000000004,"15min":13.728},"min":{"1min":6.1070000000000002,"5min":5.3449999999999998,"15min":4.5490000000000004},"max":{"1min":57.738999999999997,"5min":57.738999999999997,"15min":131.70500000000001},"last":118.562},{"interface":"front","average":{"1min":14.811,"5min":11.605,"15min":16.902000000000001},"min":{"1min":7.4320000000000004,"5min":6.0609999999999999,"15min":5.7169999999999996},"max":{"1min":60.664999999999999,"5min":60.664999999999999,"15min":202.07300000000001},"last":24.960000000000001}]},{"osd":2,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.559000000000001,"5min":15.26,"15min":18.596},"min":{"1min":8.6720000000000006,"5min":7.7460000000000004,"15min":6.9470000000000001},"max":{"1min":109.023,"5min":109.023,"15min":203.67699999999999},"last":23.725000000000001},{"interface":"front","average":{"1min":23.201000000000001,"5min":15.288,"15min":19.117000000000001},"min":{"1min":9.0129999999999999,"5min":7.8760000000000003,"15min":6.8719999999999999},"max":{"1min":93.819999999999993,"5min":93.819999999999993,"15min":211.67500000000001},"last":46.896000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":15.050000000000001,"15min":19.510000000000002},"min":{"1min":8.3170000000000002,"5min":8.0700000000000003,"15min":7.548},"max":{"1min":58.795000000000002,"5min":58.795000000000002,"15min":167.095},"last":22.346},{"interface":"front","average":{"1min":19.649000000000001,"5min":15.371,"15min":19.989000000000001},"min":{"1min":10.052,"5min":8.1690000000000005,"15min":7.6769999999999996},"max":{"1min":68.209999999999994,"5min":68.209999999999994,"15min":207.58799999999999},"last":10.401999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.547000000000001,"5min":16.661000000000001,"15min":19.132000000000001},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":6.8360000000000003},"max":{"1min":69.195999999999998,"5min":69.195999999999998,"15min":132.64500000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":33.280999999999999,"5min":20.861000000000001,"15min":23.035},"min":{"1min":14.875999999999999,"5min":9.9390000000000001,"15min":8.2590000000000003},"max":{"1min":199.828,"5min":199.828,"15min":199.828},"last":20.135999999999999}]},{"osd":6,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":23.556999999999999,"5min":12.887,"15min":16.77},"min":{"1min":6.3620000000000001,"5min":6.234,"15min":5.8789999999999996},"max":{"1min":199.03800000000001,"5min":199.03800000000001,"15min":199.03800000000001},"last":119.57299999999999},{"interface":"front","average":{"1min":26.257999999999999,"5min":19.350999999999999,"15min":19.757000000000001},"min":{"1min":11.548999999999999,"5min":11.548999999999999,"15min":6.1479999999999997},"max":{"1min":113.911,"5min":113.911,"15min":268.072},"last":111.997}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":22.385000000000002,"5min":19.629999999999999,"15min":22.059999999999999},"min":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":8.3239999999999998},"max":{"1min":63.686999999999998,"5min":63.686999999999998,"15min":198.304},"last":20.847000000000001},{"interface":"front","average":{"1min":19.053000000000001,"5min":15.726000000000001,"15min":20.466000000000001},"min":{"1min":10.156000000000001,"5min":8.0630000000000006,"15min":8.0630000000000006},"max":{"1min":56.078000000000003,"5min":56.078000000000003,"15min":181.84299999999999},"last":7.7069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-11T20:49:03.669 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-11T20:49:03.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-11T20:49:03.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.770+0000 7f51d2b65640 10 monclient: get_monmap_and_config 2023-12-11T20:49:03.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.770+0000 7f51d2b65640 10 monclient: build_initial_monmap 2023-12-11T20:49:03.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.770+0000 7f51d2b65640 10 monclient: monmap: 2023-12-11T20:49:03.769 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:03.769 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:03.771258+0000 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:03.771258+0000 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:03.770 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.771+0000 7f51d2b65640 1 Processor -- start 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.771+0000 7f51d2b65640 1 -- start start 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.771+0000 7f51d2b65640 10 monclient: init 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51cc0b0b80 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc071990 0x7f51cc143240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient(hunting): picked mon.noname-b con 0x7f51cc071990 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:03.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f51cc143780 0x7f51cc14a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient(hunting): picked mon.noname-c con 0x7f51cc143780 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51cc0a8b70 con 0x7f51cc0b0b80 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51cc072540 con 0x7f51cc143780 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc071dd0 con 0x7f51cc071990 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d2b65640 10 monclient(hunting): authenticate will time out at 3430.101372s 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51cbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc071990 0x7f51cc143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d0901640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f51cc0b0b80 0x7f51cc13d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41792/0 (socket says 172.21.15.86:41792) 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d0901640 1 -- 172.21.15.86:0/1356140693 learned_addr learned my addr 172.21.15.86:0/1356140693 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.772+0000 7f51d1102640 1 --2- 172.21.15.86:0/1356140693 >> v2:172.21.15.133:3300/0 conn(0x7f51cc143780 0x7f51cc14a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51cbfff640 10 monclient(hunting): get_auth_request con 0x7f51cc071990 auth_method 0 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51cbfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51cbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51d1102640 10 monclient(hunting): get_auth_request con 0x7f51cc143780 auth_method 0 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51d1102640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51d1102640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51d1102640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.775 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51d1102640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51d1102640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.773+0000 7f51d1102640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 10 monclient(hunting): handle_auth_done global_id 21007 payload 274 2023-12-11T20:49:03.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 1 -- 172.21.15.86:0/1356140693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc071990 msgr2=0x7f51cc143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:03.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 1 --2- 172.21.15.86:0/1356140693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc071990 0x7f51cc143240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 1 -- 172.21.15.86:0/1356140693 >> v1:172.21.15.86:6789/0 conn(0x7f51cc0b0b80 legacy=0x7f51cc13d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 10 monclient: _finish_hunting 0 2023-12-11T20:49:03.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 1 monclient: found mon.noname-c 2023-12-11T20:49:03.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:03.777 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.778 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.775+0000 7f51d1102640 1 -- 172.21.15.86:0/1356140693 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc0a8dc0 con 0x7f51cc143780 2023-12-11T20:49:03.778 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.777+0000 7f51d1102640 1 --2- 172.21.15.86:0/1356140693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc14a5d0 secure :-1 s=READY pgs=5106 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009d90 tx=0x7f51c402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11d33d72bf81f02e server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:03.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 1 -- 172.21.15.86:0/1356140693 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c402cbc0 con 0x7f51cc143780 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: dump: 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:03.781 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:03.782 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:03.782 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:03.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:03.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:03.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 1 -- 172.21.15.86:0/1356140693 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c402cd20 con 0x7f51cc143780 2023-12-11T20:49:03.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51d2b65640 5 monclient: authenticate success, global_id 21007 2023-12-11T20:49:03.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:03.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51d2b65640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:03.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51d2b65640 4 set_mon_vals no callback set 2023-12-11T20:49:03.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 1 -- 172.21.15.86:0/1356140693 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c4034660 con 0x7f51cc143780 2023-12-11T20:49:03.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:03.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:03.786 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51cb7fe640 10 monclient: dump: 2023-12-11T20:49:03.786 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:03.786 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.786 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.786 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51d2b65640 10 monclient: shutdown 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51d2b65640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51d2b65640 1 -- 172.21.15.86:0/1356140693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 msgr2=0x7f51cc14a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.781+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1356140693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc14a5d0 secure :-1 s=READY pgs=5106 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009d90 tx=0x7f51c402be10 comp rx=0 tx=0).stop 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 1 -- 172.21.15.86:0/1356140693 shutdown_connections 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1356140693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc071990 0x7f51cc143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1356140693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc14a5d0 unknown :-1 s=CLOSED pgs=5106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 1 -- 172.21.15.86:0/1356140693 >> 172.21.15.86:0/1356140693 conn(0x7f51cc0b3ef0 msgr2=0x7f51cc0b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 1 -- 172.21.15.86:0/1356140693 shutdown_connections 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 1 -- 172.21.15.86:0/1356140693 wait complete. 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 10 monclient: build_initial_monmap 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 10 monclient: monmap: 2023-12-11T20:49:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:03.789 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 1 librados: starting msgr at 2023-12-11T20:49:03.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.782+0000 7f51d2b65640 1 librados: starting objecter 2023-12-11T20:49:03.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 Processor -- start 2023-12-11T20:49:03.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 -- start start 2023-12-11T20:49:03.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 librados: setting wanted keys 2023-12-11T20:49:03.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 librados: calling monclient init 2023-12-11T20:49:03.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient: init 2023-12-11T20:49:03.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:03.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:49:03.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51cc071990 0x7f51cc0a9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient(hunting): picked mon.noname-a con 0x7f51cc071990 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:03.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc0b0b80 0x7f51cc135b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient(hunting): picked mon.noname-c con 0x7f51cc0b0b80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:03.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient(hunting): picked mon.noname-b con 0x7f51cc143780 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:03.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0aa210 con 0x7f51cc071990 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc136580 con 0x7f51cc143780 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc13dc00 con 0x7f51cc0b0b80 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.783+0000 7f51d2b65640 10 monclient(hunting): authenticate will time out at 3430.112990s 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51cbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc0b0b80 0x7f51cc135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d0901640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51cc071990 0x7f51cc0a9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc136040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:55454/0 (socket says 172.21.15.86:55454) 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 1 -- 172.21.15.86:0/1667322683 learned_addr learned my addr 172.21.15.86:0/1667322683 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 10 monclient(hunting): get_auth_request con 0x7f51cc143780 auth_method 0 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51cbfff640 10 monclient(hunting): get_auth_request con 0x7f51cc0b0b80 auth_method 0 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51cbfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51cbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d0901640 10 monclient(hunting): get_auth_request con 0x7f51cc071990 auth_method 0 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d0901640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d0901640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d0901640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.784+0000 7f51d1102640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.785+0000 7f51cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.785+0000 7f51cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.785+0000 7f51cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.785+0000 7f51d0901640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.785+0000 7f51d0901640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.785+0000 7f51d0901640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 10 monclient(hunting): handle_auth_done global_id 21010 payload 824 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 1 -- 172.21.15.86:0/1667322683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc0b0b80 msgr2=0x7f51cc135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc0b0b80 0x7f51cc135b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 1 -- 172.21.15.86:0/1667322683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51cc071990 msgr2=0x7f51cc0a9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51cc071990 0x7f51cc0a9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 10 monclient: _finish_hunting 0 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 1 monclient: found mon.noname-b 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.787+0000 7f51d1102640 1 -- 172.21.15.86:0/1667322683 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c4009a70 con 0x7f51cc143780 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.789+0000 7f51d1102640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc136040 secure :-1 s=READY pgs=5107 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009eb0 tx=0x7f51c4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.789+0000 7f51c97fa640 10 client.?.objecter ms_handle_connect 0x7f51cc143780 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.789+0000 7f51c97fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 1 -- 172.21.15.86:0/1667322683 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c4040650 con 0x7f51cc143780 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: dump: 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: _finish_auth 0 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: _check_auth_tickets 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 1 -- 172.21.15.86:0/1667322683 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c402ce90 con 0x7f51cc143780 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 5 monclient: authenticate success, global_id 21010 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 10 monclient: _renew_subs 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc13d950 con 0x7f51cc143780 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 1 -- 172.21.15.86:0/1667322683 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c40409f0 con 0x7f51cc143780 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51c97fa640 10 monclient: dump: 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:03.804 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51ca7fc640 4 set_mon_vals no callback set 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 10 client.21010.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 10 monclient: _renew_subs 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc140af0 con 0x7f51cc143780 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.793+0000 7f51d2b65640 1 librados: init done 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.794+0000 7f51d2b65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.794+0000 7f51d2b65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.794+0000 7f51d2b65640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.794+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51cc1d4cc0 con 0x7f51cc143780 2023-12-11T20:49:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.812+0000 7f51c97fa640 1 -- 172.21.15.86:0/1667322683 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f51c4040b50 con 0x7f51cc143780 2023-12-11T20:49:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.813+0000 7f51c97fa640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f51b806c870 0x7f51b806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.813+0000 7f51d0901640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f51b806c870 0x7f51b806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.813+0000 7f51d0901640 10 monclient: get_auth_request con 0x7f51b806c870 auth_method 0 2023-12-11T20:49:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.814+0000 7f51d0901640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f51b806c870 0x7f51b806ed30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f51bc007fd0 tx=0x7f51bc007e80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:03.814+0000 7f51c97fa640 10 client.21010.objecter ms_handle_connect 0x7f51b806c870 2023-12-11T20:49:04.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.033+0000 7f51c97fa640 1 -- 172.21.15.86:0/1667322683 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f51c407b1b0 con 0x7f51cc143780 2023-12-11T20:49:04.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.033+0000 7f51c97fa640 10 client.21010.objecter ms_dispatch 0x7f51cc000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:04.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.033+0000 7f51c97fa640 3 client.21010.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:04.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.033+0000 7f51c97fa640 3 client.21010.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:04.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.034+0000 7f51c97fa640 20 client.21010.objecter dump_active .. 0 homeless 2023-12-11T20:49:04.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.044+0000 7f51c97fa640 1 -- 172.21.15.86:0/1667322683 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51c40774f0 con 0x7f51cc143780 2023-12-11T20:49:04.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.044+0000 7f51c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:04.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.044+0000 7f51c97fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:04.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.237+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51cc0729f0 con 0x7f51b806c870 2023-12-11T20:49:04.242 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.242+0000 7f51c97fa640 1 -- 172.21.15.86:0/1667322683 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151646 (secure 0 0 0) 0x7f51bc001460 con 0x7f51b806c870 2023-12-11T20:49:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:49:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-11T20:49:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.244+0000 7f51d2b65640 10 librados: watch_flush enter 2023-12-11T20:49:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.244+0000 7f51d2b65640 10 librados: watch_flush exit 2023-12-11T20:49:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 20 client.21010.objecter shutdown clearing up homeless session... 2023-12-11T20:49:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 10 client.21010.objecter successfully canceled tick 2023-12-11T20:49:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f51b806c870 msgr2=0x7f51b806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:04.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f51b806c870 0x7f51b806ed30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f51bc007fd0 tx=0x7f51bc007e80 comp rx=0 tx=0).stop 2023-12-11T20:49:04.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 10 monclient: shutdown 2023-12-11T20:49:04.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:04.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 msgr2=0x7f51cc136040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:04.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc136040 secure :-1 s=READY pgs=5107 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009eb0 tx=0x7f51c4004a40 comp rx=0 tx=0).stop 2023-12-11T20:49:04.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 shutdown_connections 2023-12-11T20:49:04.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f51b806c870 0x7f51b806ed30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51cc0b0b80 0x7f51cc135b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51cc143780 0x7f51cc136040 unknown :-1 s=CLOSED pgs=5107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 --2- 172.21.15.86:0/1667322683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51cc071990 0x7f51cc0a9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 >> 172.21.15.86:0/1667322683 conn(0x7f51cc0b3ef0 msgr2=0x7f51cc13ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:04.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 shutdown_connections 2023-12-11T20:49:04.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 -- 172.21.15.86:0/1667322683 wait complete. 2023-12-11T20:49:04.253 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.245+0000 7f51d2b65640 1 librados: shutdown 2023-12-11T20:49:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":2030,"stamp":"2023-12-11T20:49:03.109616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":534868,"kb_used_data":16964,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754439852,"statfs":{"total":773094113280,"available":772546408448,"internally_reserved":0,"allocated":17371136,"data_stored":5109063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":529714737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":154,"apply_latency_ms":154,"commit_latency_ns":154000000,"apply_latency_ns":154000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.630347"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1000,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.493525+0000","last_change":"2023-12-11T20:18:00.538563+0000","last_active":"2023-12-11T20:48:58.493525+0000","last_peered":"2023-12-11T20:48:58.493525+0000","last_clean":"2023-12-11T20:48:58.493525+0000","last_became_active":"2023-12-11T20:18:00.530681+0000","last_became_peered":"2023-12-11T20:18:00.530681+0000","last_unstale":"2023-12-11T20:48:58.493525+0000","last_undegraded":"2023-12-11T20:48:58.493525+0000","last_fullsized":"2023-12-11T20:48:58.493525+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:23:20.646623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1006,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.387229+0000","last_change":"2023-12-11T20:18:01.470212+0000","last_active":"2023-12-11T20:48:57.387229+0000","last_peered":"2023-12-11T20:48:57.387229+0000","last_clean":"2023-12-11T20:48:57.387229+0000","last_became_active":"2023-12-11T20:18:01.459740+0000","last_became_peered":"2023-12-11T20:18:01.459740+0000","last_unstale":"2023-12-11T20:48:57.387229+0000","last_undegraded":"2023-12-11T20:48:57.387229+0000","last_fullsized":"2023-12-11T20:48:57.387229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:33:49.955305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1011,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.877928+0000","last_change":"2023-12-11T20:18:11.560096+0000","last_active":"2023-12-11T20:49:00.877928+0000","last_peered":"2023-12-11T20:49:00.877928+0000","last_clean":"2023-12-11T20:49:00.877928+0000","last_became_active":"2023-12-11T20:18:11.552993+0000","last_became_peered":"2023-12-11T20:18:11.552993+0000","last_unstale":"2023-12-11T20:49:00.877928+0000","last_undegraded":"2023-12-11T20:49:00.877928+0000","last_fullsized":"2023-12-11T20:49:00.877928+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:31:04.664827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1002,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.093452+0000","last_change":"2023-12-11T20:18:04.754752+0000","last_active":"2023-12-11T20:48:57.093452+0000","last_peered":"2023-12-11T20:48:57.093452+0000","last_clean":"2023-12-11T20:48:57.093452+0000","last_became_active":"2023-12-11T20:18:04.748213+0000","last_became_peered":"2023-12-11T20:18:04.748213+0000","last_unstale":"2023-12-11T20:48:57.093452+0000","last_undegraded":"2023-12-11T20:48:57.093452+0000","last_fullsized":"2023-12-11T20:48:57.093452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:42:40.065588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1005,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.107408+0000","last_change":"2023-12-11T20:18:07.107260+0000","last_active":"2023-12-11T20:48:57.107408+0000","last_peered":"2023-12-11T20:48:57.107408+0000","last_clean":"2023-12-11T20:48:57.107408+0000","last_became_active":"2023-12-11T20:18:06.871380+0000","last_became_peered":"2023-12-11T20:18:06.871380+0000","last_unstale":"2023-12-11T20:48:57.107408+0000","last_undegraded":"2023-12-11T20:48:57.107408+0000","last_fullsized":"2023-12-11T20:48:57.107408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:09:46.999561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1008,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.391051+0000","last_change":"2023-12-11T20:18:00.913863+0000","last_active":"2023-12-11T20:48:57.391051+0000","last_peered":"2023-12-11T20:48:57.391051+0000","last_clean":"2023-12-11T20:48:57.391051+0000","last_became_active":"2023-12-11T20:18:00.891399+0000","last_became_peered":"2023-12-11T20:18:00.891399+0000","last_unstale":"2023-12-11T20:48:57.391051+0000","last_undegraded":"2023-12-11T20:48:57.391051+0000","last_fullsized":"2023-12-11T20:48:57.391051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:30:57.714593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1013,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.892004+0000","last_change":"2023-12-11T20:18:11.329207+0000","last_active":"2023-12-11T20:49:00.892004+0000","last_peered":"2023-12-11T20:49:00.892004+0000","last_clean":"2023-12-11T20:49:00.892004+0000","last_became_active":"2023-12-11T20:18:11.323743+0000","last_became_peered":"2023-12-11T20:18:11.323743+0000","last_unstale":"2023-12-11T20:49:00.892004+0000","last_undegraded":"2023-12-11T20:49:00.892004+0000","last_fullsized":"2023-12-11T20:49:00.892004+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:45:34.411191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1021,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.358567+0000","last_change":"2023-12-11T20:18:03.803423+0000","last_active":"2023-12-11T20:48:57.358567+0000","last_peered":"2023-12-11T20:48:57.358567+0000","last_clean":"2023-12-11T20:48:57.358567+0000","last_became_active":"2023-12-11T20:18:03.798156+0000","last_became_peered":"2023-12-11T20:18:03.798156+0000","last_unstale":"2023-12-11T20:48:57.358567+0000","last_undegraded":"2023-12-11T20:48:57.358567+0000","last_fullsized":"2023-12-11T20:48:57.358567+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:31:17.842399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1015,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.734172+0000","last_change":"2023-12-11T20:18:06.556522+0000","last_active":"2023-12-11T20:48:58.734172+0000","last_peered":"2023-12-11T20:48:58.734172+0000","last_clean":"2023-12-11T20:48:58.734172+0000","last_became_active":"2023-12-11T20:18:06.509738+0000","last_became_peered":"2023-12-11T20:18:06.509738+0000","last_unstale":"2023-12-11T20:48:58.734172+0000","last_undegraded":"2023-12-11T20:48:58.734172+0000","last_fullsized":"2023-12-11T20:48:58.734172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T00:20:54.755785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"118'7","reported_seq":1026,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.874651+0000","last_change":"2023-12-11T20:18:05.708954+0000","last_active":"2023-12-11T20:49:00.874651+0000","last_peered":"2023-12-11T20:49:00.874651+0000","last_clean":"2023-12-11T20:49:00.874651+0000","last_became_active":"2023-12-11T20:18:05.684774+0000","last_became_peered":"2023-12-11T20:18:05.684774+0000","last_unstale":"2023-12-11T20:49:00.874651+0000","last_undegraded":"2023-12-11T20:49:00.874651+0000","last_fullsized":"2023-12-11T20:49:00.874651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:35:07.384658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1005,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.090750+0000","last_change":"2023-12-11T20:18:11.631063+0000","last_active":"2023-12-11T20:48:57.090750+0000","last_peered":"2023-12-11T20:48:57.090750+0000","last_clean":"2023-12-11T20:48:57.090750+0000","last_became_active":"2023-12-11T20:18:11.576567+0000","last_became_peered":"2023-12-11T20:18:11.576567+0000","last_unstale":"2023-12-11T20:48:57.090750+0000","last_undegraded":"2023-12-11T20:48:57.090750+0000","last_fullsized":"2023-12-11T20:48:57.090750+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:22:48.857424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1021,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.353929+0000","last_change":"2023-12-11T20:18:03.266276+0000","last_active":"2023-12-11T20:48:57.353929+0000","last_peered":"2023-12-11T20:48:57.353929+0000","last_clean":"2023-12-11T20:48:57.353929+0000","last_became_active":"2023-12-11T20:18:03.237738+0000","last_became_peered":"2023-12-11T20:18:03.237738+0000","last_unstale":"2023-12-11T20:48:57.353929+0000","last_undegraded":"2023-12-11T20:48:57.353929+0000","last_fullsized":"2023-12-11T20:48:57.353929+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:00:38.864099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":1118,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.380908+0000","last_change":"2023-12-11T20:16:49.468937+0000","last_active":"2023-12-11T20:48:57.380908+0000","last_peered":"2023-12-11T20:48:57.380908+0000","last_clean":"2023-12-11T20:48:57.380908+0000","last_became_active":"2023-12-11T20:16:49.435576+0000","last_became_peered":"2023-12-11T20:16:49.435576+0000","last_unstale":"2023-12-11T20:48:57.380908+0000","last_undegraded":"2023-12-11T20:48:57.380908+0000","last_fullsized":"2023-12-11T20:48:57.380908+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:45.844700+0000","last_clean_scrub_stamp":"2023-12-11T20:16:45.844700+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:49:09.602489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.409096+0000","last_change":"2023-12-11T20:16:58.643847+0000","last_active":"2023-12-11T20:48:57.409096+0000","last_peered":"2023-12-11T20:48:57.409096+0000","last_clean":"2023-12-11T20:48:57.409096+0000","last_became_active":"2023-12-11T20:16:58.638544+0000","last_became_peered":"2023-12-11T20:16:58.638544+0000","last_unstale":"2023-12-11T20:48:57.409096+0000","last_undegraded":"2023-12-11T20:48:57.409096+0000","last_fullsized":"2023-12-11T20:48:57.409096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:17:51.003853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1002,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.502405+0000","last_change":"2023-12-11T20:18:12.933720+0000","last_active":"2023-12-11T20:48:58.502405+0000","last_peered":"2023-12-11T20:48:58.502405+0000","last_clean":"2023-12-11T20:48:58.502405+0000","last_became_active":"2023-12-11T20:18:12.924715+0000","last_became_peered":"2023-12-11T20:18:12.924715+0000","last_unstale":"2023-12-11T20:48:58.502405+0000","last_undegraded":"2023-12-11T20:48:58.502405+0000","last_fullsized":"2023-12-11T20:48:58.502405+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:57:59.553539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1010,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.398408+0000","last_change":"2023-12-11T20:16:58.036508+0000","last_active":"2023-12-11T20:48:57.398408+0000","last_peered":"2023-12-11T20:48:57.398408+0000","last_clean":"2023-12-11T20:48:57.398408+0000","last_became_active":"2023-12-11T20:16:58.008642+0000","last_became_peered":"2023-12-11T20:16:58.008642+0000","last_unstale":"2023-12-11T20:48:57.398408+0000","last_undegraded":"2023-12-11T20:48:57.398408+0000","last_fullsized":"2023-12-11T20:48:57.398408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:46:42.050513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1015,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.742328+0000","last_change":"2023-12-11T20:18:07.144229+0000","last_active":"2023-12-11T20:48:58.742328+0000","last_peered":"2023-12-11T20:48:58.742328+0000","last_clean":"2023-12-11T20:48:58.742328+0000","last_became_active":"2023-12-11T20:18:07.135656+0000","last_became_peered":"2023-12-11T20:18:07.135656+0000","last_unstale":"2023-12-11T20:48:58.742328+0000","last_undegraded":"2023-12-11T20:48:58.742328+0000","last_fullsized":"2023-12-11T20:48:58.742328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:48:10.044525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1026,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.344703+0000","last_change":"2023-12-11T20:16:58.236956+0000","last_active":"2023-12-11T20:48:57.344703+0000","last_peered":"2023-12-11T20:48:57.344703+0000","last_clean":"2023-12-11T20:48:57.344703+0000","last_became_active":"2023-12-11T20:16:58.208987+0000","last_became_peered":"2023-12-11T20:16:58.208987+0000","last_unstale":"2023-12-11T20:48:57.344703+0000","last_undegraded":"2023-12-11T20:48:57.344703+0000","last_fullsized":"2023-12-11T20:48:57.344703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:57:37.650191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1023,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.237706+0000","last_change":"2023-12-11T20:18:01.362777+0000","last_active":"2023-12-11T20:49:00.237706+0000","last_peered":"2023-12-11T20:49:00.237706+0000","last_clean":"2023-12-11T20:49:00.237706+0000","last_became_active":"2023-12-11T20:18:01.354083+0000","last_became_peered":"2023-12-11T20:18:01.354083+0000","last_unstale":"2023-12-11T20:49:00.237706+0000","last_undegraded":"2023-12-11T20:49:00.237706+0000","last_fullsized":"2023-12-11T20:49:00.237706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:40:18.542456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.411468+0000","last_change":"2023-12-11T20:16:58.356627+0000","last_active":"2023-12-11T20:48:57.411468+0000","last_peered":"2023-12-11T20:48:57.411468+0000","last_clean":"2023-12-11T20:48:57.411468+0000","last_became_active":"2023-12-11T20:16:58.349599+0000","last_became_peered":"2023-12-11T20:16:58.349599+0000","last_unstale":"2023-12-11T20:48:57.411468+0000","last_undegraded":"2023-12-11T20:48:57.411468+0000","last_fullsized":"2023-12-11T20:48:57.411468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:45:43.507550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1007,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.382641+0000","last_change":"2023-12-11T20:18:02.135407+0000","last_active":"2023-12-11T20:48:57.382641+0000","last_peered":"2023-12-11T20:48:57.382641+0000","last_clean":"2023-12-11T20:48:57.382641+0000","last_became_active":"2023-12-11T20:18:02.117789+0000","last_became_peered":"2023-12-11T20:18:02.117789+0000","last_unstale":"2023-12-11T20:48:57.382641+0000","last_undegraded":"2023-12-11T20:48:57.382641+0000","last_fullsized":"2023-12-11T20:48:57.382641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:59:14.324843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1020,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.752807+0000","last_change":"2023-12-11T20:16:58.412827+0000","last_active":"2023-12-11T20:48:58.752807+0000","last_peered":"2023-12-11T20:48:58.752807+0000","last_clean":"2023-12-11T20:48:58.752807+0000","last_became_active":"2023-12-11T20:16:58.383357+0000","last_became_peered":"2023-12-11T20:16:58.383357+0000","last_unstale":"2023-12-11T20:48:58.752807+0000","last_undegraded":"2023-12-11T20:48:58.752807+0000","last_fullsized":"2023-12-11T20:48:58.752807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T20:59:10.927708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1014,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.765129+0000","last_change":"2023-12-11T20:18:13.718317+0000","last_active":"2023-12-11T20:48:58.765129+0000","last_peered":"2023-12-11T20:48:58.765129+0000","last_clean":"2023-12-11T20:48:58.765129+0000","last_became_active":"2023-12-11T20:18:13.700424+0000","last_became_peered":"2023-12-11T20:18:13.700424+0000","last_unstale":"2023-12-11T20:48:58.765129+0000","last_undegraded":"2023-12-11T20:48:58.765129+0000","last_fullsized":"2023-12-11T20:48:58.765129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:57:09.221399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1011,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.428322+0000","last_change":"2023-12-11T20:16:58.680128+0000","last_active":"2023-12-11T20:48:57.428322+0000","last_peered":"2023-12-11T20:48:57.428322+0000","last_clean":"2023-12-11T20:48:57.428322+0000","last_became_active":"2023-12-11T20:16:58.674889+0000","last_became_peered":"2023-12-11T20:16:58.674889+0000","last_unstale":"2023-12-11T20:48:57.428322+0000","last_undegraded":"2023-12-11T20:48:57.428322+0000","last_fullsized":"2023-12-11T20:48:57.428322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:54:50.773022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1020,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.357034+0000","last_change":"2023-12-11T20:18:03.189633+0000","last_active":"2023-12-11T20:48:57.357034+0000","last_peered":"2023-12-11T20:48:57.357034+0000","last_clean":"2023-12-11T20:48:57.357034+0000","last_became_active":"2023-12-11T20:18:03.180691+0000","last_became_peered":"2023-12-11T20:18:03.180691+0000","last_unstale":"2023-12-11T20:48:57.357034+0000","last_undegraded":"2023-12-11T20:48:57.357034+0000","last_fullsized":"2023-12-11T20:48:57.357034+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:26:16.797482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1028,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.281191+0000","last_change":"2023-12-11T20:16:57.283249+0000","last_active":"2023-12-11T20:49:00.281191+0000","last_peered":"2023-12-11T20:49:00.281191+0000","last_clean":"2023-12-11T20:49:00.281191+0000","last_became_active":"2023-12-11T20:16:57.255904+0000","last_became_peered":"2023-12-11T20:16:57.255904+0000","last_unstale":"2023-12-11T20:49:00.281191+0000","last_undegraded":"2023-12-11T20:49:00.281191+0000","last_fullsized":"2023-12-11T20:49:00.281191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:17:34.118238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1014,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.749638+0000","last_change":"2023-12-11T20:18:33.771824+0000","last_active":"2023-12-11T20:48:58.749638+0000","last_peered":"2023-12-11T20:48:58.749638+0000","last_clean":"2023-12-11T20:48:58.749638+0000","last_became_active":"2023-12-11T20:18:33.762733+0000","last_became_peered":"2023-12-11T20:18:33.762733+0000","last_unstale":"2023-12-11T20:48:58.749638+0000","last_undegraded":"2023-12-11T20:48:58.749638+0000","last_fullsized":"2023-12-11T20:48:58.749638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:53.954274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1010,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.424106+0000","last_change":"2023-12-11T20:16:58.076986+0000","last_active":"2023-12-11T20:48:57.424106+0000","last_peered":"2023-12-11T20:48:57.424106+0000","last_clean":"2023-12-11T20:48:57.424106+0000","last_became_active":"2023-12-11T20:16:58.071647+0000","last_became_peered":"2023-12-11T20:16:58.071647+0000","last_unstale":"2023-12-11T20:48:57.424106+0000","last_undegraded":"2023-12-11T20:48:57.424106+0000","last_fullsized":"2023-12-11T20:48:57.424106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:29:07.975896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1003,"reported_epoch":1052,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.103446+0000","last_change":"2023-12-11T20:18:11.891380+0000","last_active":"2023-12-11T20:48:57.103446+0000","last_peered":"2023-12-11T20:48:57.103446+0000","last_clean":"2023-12-11T20:48:57.103446+0000","last_became_active":"2023-12-11T20:18:11.885388+0000","last_became_peered":"2023-12-11T20:18:11.885388+0000","last_unstale":"2023-12-11T20:48:57.103446+0000","last_undegraded":"2023-12-11T20:48:57.103446+0000","last_fullsized":"2023-12-11T20:48:57.103446+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:37:07.253751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1024,"reported_epoch":1055,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.284857+0000","last_change":"2023-12-11T20:18:12.794888+0000","last_active":"2023-12-11T20:49:00.284857+0000","last_peered":"2023-12-11T20:49:00.284857+0000","last_clean":"2023-12-11T20:49:00.284857+0000","last_became_active":"2023-12-11T20:18:12.786107+0000","last_became_peered":"2023-12-11T20:18:12.786107+0000","last_unstale":"2023-12-11T20:49:00.284857+0000","last_undegraded":"2023-12-11T20:49:00.284857+0000","last_fullsized":"2023-12-11T20:49:00.284857+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:00:25.310704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1022,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.360301+0000","last_change":"2023-12-11T20:18:00.474665+0000","last_active":"2023-12-11T20:48:57.360301+0000","last_peered":"2023-12-11T20:48:57.360301+0000","last_clean":"2023-12-11T20:48:57.360301+0000","last_became_active":"2023-12-11T20:18:00.469176+0000","last_became_peered":"2023-12-11T20:18:00.469176+0000","last_unstale":"2023-12-11T20:48:57.360301+0000","last_undegraded":"2023-12-11T20:48:57.360301+0000","last_fullsized":"2023-12-11T20:48:57.360301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:24:10.853739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1011,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.886618+0000","last_change":"2023-12-11T20:18:05.144024+0000","last_active":"2023-12-11T20:49:00.886618+0000","last_peered":"2023-12-11T20:49:00.886618+0000","last_clean":"2023-12-11T20:49:00.886618+0000","last_became_active":"2023-12-11T20:18:05.138035+0000","last_became_peered":"2023-12-11T20:18:05.138035+0000","last_unstale":"2023-12-11T20:49:00.886618+0000","last_undegraded":"2023-12-11T20:49:00.886618+0000","last_fullsized":"2023-12-11T20:49:00.886618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:20:37.637313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.416205+0000","last_change":"2023-12-11T20:18:10.100025+0000","last_active":"2023-12-11T20:48:57.416205+0000","last_peered":"2023-12-11T20:48:57.416205+0000","last_clean":"2023-12-11T20:48:57.416205+0000","last_became_active":"2023-12-11T20:18:10.094694+0000","last_became_peered":"2023-12-11T20:18:10.094694+0000","last_unstale":"2023-12-11T20:48:57.416205+0000","last_undegraded":"2023-12-11T20:48:57.416205+0000","last_fullsized":"2023-12-11T20:48:57.416205+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:38:01.652138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1013,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.745464+0000","last_change":"2023-12-11T20:18:07.073119+0000","last_active":"2023-12-11T20:48:58.745464+0000","last_peered":"2023-12-11T20:48:58.745464+0000","last_clean":"2023-12-11T20:48:58.745464+0000","last_became_active":"2023-12-11T20:18:07.062338+0000","last_became_peered":"2023-12-11T20:18:07.062338+0000","last_unstale":"2023-12-11T20:48:58.745464+0000","last_undegraded":"2023-12-11T20:48:58.745464+0000","last_fullsized":"2023-12-11T20:48:58.745464+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:48:41.166067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1005,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.413077+0000","last_change":"2023-12-11T20:18:19.678310+0000","last_active":"2023-12-11T20:48:57.413077+0000","last_peered":"2023-12-11T20:48:57.413077+0000","last_clean":"2023-12-11T20:48:57.413077+0000","last_became_active":"2023-12-11T20:18:19.665316+0000","last_became_peered":"2023-12-11T20:18:19.665316+0000","last_unstale":"2023-12-11T20:48:57.413077+0000","last_undegraded":"2023-12-11T20:48:57.413077+0000","last_fullsized":"2023-12-11T20:48:57.413077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:23:40.772081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1005,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.405963+0000","last_change":"2023-12-11T20:18:18.565871+0000","last_active":"2023-12-11T20:48:57.405963+0000","last_peered":"2023-12-11T20:48:57.405963+0000","last_clean":"2023-12-11T20:48:57.405963+0000","last_became_active":"2023-12-11T20:18:18.557249+0000","last_became_peered":"2023-12-11T20:18:18.557249+0000","last_unstale":"2023-12-11T20:48:57.405963+0000","last_undegraded":"2023-12-11T20:48:57.405963+0000","last_fullsized":"2023-12-11T20:48:57.405963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:35:50.576838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.402112+0000","last_change":"2023-12-11T20:18:19.693007+0000","last_active":"2023-12-11T20:48:57.402112+0000","last_peered":"2023-12-11T20:48:57.402112+0000","last_clean":"2023-12-11T20:48:57.402112+0000","last_became_active":"2023-12-11T20:18:19.685187+0000","last_became_peered":"2023-12-11T20:18:19.685187+0000","last_unstale":"2023-12-11T20:48:57.402112+0000","last_undegraded":"2023-12-11T20:48:57.402112+0000","last_fullsized":"2023-12-11T20:48:57.402112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:01:40.810139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1006,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.395902+0000","last_change":"2023-12-11T20:18:00.559982+0000","last_active":"2023-12-11T20:48:57.395902+0000","last_peered":"2023-12-11T20:48:57.395902+0000","last_clean":"2023-12-11T20:48:57.395902+0000","last_became_active":"2023-12-11T20:18:00.554682+0000","last_became_peered":"2023-12-11T20:18:00.554682+0000","last_unstale":"2023-12-11T20:48:57.395902+0000","last_undegraded":"2023-12-11T20:48:57.395902+0000","last_fullsized":"2023-12-11T20:48:57.395902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:12:05.725839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1013,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.763610+0000","last_change":"2023-12-11T20:18:13.715085+0000","last_active":"2023-12-11T20:48:58.763610+0000","last_peered":"2023-12-11T20:48:58.763610+0000","last_clean":"2023-12-11T20:48:58.763610+0000","last_became_active":"2023-12-11T20:18:13.689623+0000","last_became_peered":"2023-12-11T20:18:13.689623+0000","last_unstale":"2023-12-11T20:48:58.763610+0000","last_undegraded":"2023-12-11T20:48:58.763610+0000","last_fullsized":"2023-12-11T20:48:58.763610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:14:13.791518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1004,"reported_epoch":1053,"state":"active+clean","last_fresh":"2023-12-11T20:48:57.421139+0000","last_change":"2023-12-11T20:18:10.128111+0000","last_active":"2023-12-11T20:48:57.421139+0000","last_peered":"2023-12-11T20:48:57.421139+0000","last_clean":"2023-12-11T20:48:57.421139+0000","last_became_active":"2023-12-11T20:18:10.112911+0000","last_became_peered":"2023-12-11T20:18:10.112911+0000","last_unstale":"2023-12-11T20:48:57.421139+0000","last_undegraded":"2023-12-11T20:48:57.421139+0000","last_fullsized":"2023-12-11T20:48:57.421139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:55:06.836498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1000,"reported_epoch":1054,"state":"active+clean","last_fresh":"2023-12-11T20:48:58.509001+0000","last_change":"2023-12-11T20:18:13.209324+0000","last_active":"2023-12-11T20:48:58.509001+0000","last_peered":"2023-12-11T20:48:58.509001+0000","last_clean":"2023-12-11T20:48:58.509001+0000","last_became_active":"2023-12-11T20:18:13.203517+0000","last_became_peered":"2023-12-11T20:18:13.203517+0000","last_unstale":"2023-12-11T20:48:58.509001+0000","last_undegraded":"2023-12-11T20:48:58.509001+0000","last_fullsized":"2023-12-11T20:48:58.509001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:09:42.156893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719477124,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69332,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302508,"statfs":{"total":96636764160,"available":96565768192,"internally_reserved":0,"allocated":2105344,"data_stored":556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":17.427,"5min":10.535,"15min":14.893000000000001},"min":{"1min":4.5739999999999998,"5min":4.5739999999999998,"15min":4.2510000000000003},"max":{"1min":112.848,"5min":112.848,"15min":221.21299999999999},"last":5.3890000000000002},{"interface":"front","average":{"1min":17.669,"5min":11.170999999999999,"15min":14.507999999999999},"min":{"1min":6.2409999999999997,"5min":5.3739999999999997,"15min":4.7930000000000001},"max":{"1min":94.400999999999996,"5min":94.400999999999996,"15min":164.01900000000001},"last":6.2089999999999996}]},{"osd":1,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":21.391999999999999,"5min":12.878,"15min":16.920999999999999},"min":{"1min":7.6660000000000004,"5min":6.8419999999999996,"15min":5.0380000000000003},"max":{"1min":117.63,"5min":117.63,"15min":195.87200000000001},"last":9.2149999999999999},{"interface":"front","average":{"1min":26.209,"5min":17.108000000000001,"15min":20.576000000000001},"min":{"1min":10.843999999999999,"5min":9.6649999999999991,"15min":6.0300000000000002},"max":{"1min":118.309,"5min":118.309,"15min":190.53899999999999},"last":10.846}]},{"osd":2,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":15.789999999999999,"5min":11.457000000000001,"15min":18.841999999999999},"min":{"1min":6.4089999999999998,"5min":6.4089999999999998,"15min":5.8259999999999996},"max":{"1min":67.793000000000006,"5min":67.793000000000006,"15min":282.79000000000002},"last":11.608000000000001},{"interface":"front","average":{"1min":21.603000000000002,"5min":15.961,"15min":21.614000000000001},"min":{"1min":8.1829999999999998,"5min":8.1829999999999998,"15min":8.0570000000000004},"max":{"1min":68.638999999999996,"5min":68.638999999999996,"15min":200.148},"last":12.178000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.715,"5min":15.484999999999999,"15min":19.199999999999999},"min":{"1min":9.6660000000000004,"5min":6.5270000000000001,"15min":5.3029999999999999},"max":{"1min":113.89400000000001,"5min":113.89400000000001,"15min":206.11799999999999},"last":12.895},{"interface":"front","average":{"1min":22.518000000000001,"5min":14.561,"15min":20.105},"min":{"1min":6.7809999999999997,"5min":6.7809999999999997,"15min":6.7809999999999997},"max":{"1min":140.172,"5min":140.172,"15min":171.63},"last":14.06}]},{"osd":4,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":24.693000000000001,"5min":16.344999999999999,"15min":18.094000000000001},"min":{"1min":8.5180000000000007,"5min":7.6289999999999996,"15min":6.577},"max":{"1min":110.98399999999999,"5min":110.98399999999999,"15min":188.26599999999999},"last":15.808},{"interface":"front","average":{"1min":25.422999999999998,"5min":17.34,"15min":21.085000000000001},"min":{"1min":7.9279999999999999,"5min":7.8470000000000004,"15min":7.5300000000000002},"max":{"1min":144.084,"5min":144.084,"15min":172.57400000000001},"last":17.742999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":19.388999999999999,"5min":14.335000000000001,"15min":18.338999999999999},"min":{"1min":8.4640000000000004,"5min":7.4800000000000004,"15min":6.141},"max":{"1min":62.637,"5min":62.637,"15min":201.25399999999999},"last":14.875999999999999},{"interface":"front","average":{"1min":24.920999999999999,"5min":17.582999999999998,"15min":21.760000000000002},"min":{"1min":10.914999999999999,"5min":8.9529999999999994,"15min":8.423},"max":{"1min":92.804000000000002,"5min":92.804000000000002,"15min":200.60599999999999},"last":16.766999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.809999999999999,"5min":18.367999999999999,"15min":21.443000000000001},"min":{"1min":10.18,"5min":8.4909999999999997,"15min":7.899},"max":{"1min":76.634,"5min":76.634,"15min":207.72200000000001},"last":38.551000000000002},{"interface":"front","average":{"1min":22.077000000000002,"5min":17.238,"15min":22.446999999999999},"min":{"1min":9.6129999999999995,"5min":9.3019999999999996,"15min":9.3019999999999996},"max":{"1min":69.144000000000005,"5min":69.144000000000005,"15min":193.43899999999999},"last":25.155999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477123,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65128,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306712,"statfs":{"total":96636764160,"available":96570073088,"internally_reserved":0,"allocated":2060288,"data_stored":534065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":21.152000000000001,"5min":11.818,"15min":15.324999999999999},"min":{"1min":6.3159999999999998,"5min":4.4720000000000004,"15min":4.1600000000000001},"max":{"1min":102.907,"5min":102.907,"15min":266.75400000000002},"last":11.272},{"interface":"front","average":{"1min":22.907,"5min":12.157999999999999,"15min":16.690999999999999},"min":{"1min":5.4100000000000001,"5min":5.4100000000000001,"15min":4.5330000000000004},"max":{"1min":110.068,"5min":110.068,"15min":228.12899999999999},"last":9.0730000000000004}]},{"osd":1,"last update":"Mon Dec 11 20:48:39 2023","interfaces":[{"interface":"back","average":{"1min":42.024000000000001,"5min":17.748999999999999,"15min":16.431000000000001},"min":{"1min":7.1529999999999996,"5min":6.7489999999999997,"15min":5.6180000000000003},"max":{"1min":170.31200000000001,"5min":170.31200000000001,"15min":179.11000000000001},"last":10.391},{"interface":"front","average":{"1min":50.826000000000001,"5min":19.158999999999999,"15min":18.561},"min":{"1min":8.9849999999999994,"5min":7.0529999999999999,"15min":6.508},"max":{"1min":215.578,"5min":215.578,"15min":215.578},"last":13.814}]},{"osd":2,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":27.824000000000002,"5min":15.458,"15min":18.983000000000001},"min":{"1min":8.6820000000000004,"5min":8.1479999999999997,"15min":7.1150000000000002},"max":{"1min":146.12700000000001,"5min":146.12700000000001,"15min":226.56299999999999},"last":9.423},{"interface":"front","average":{"1min":30.817,"5min":17.829000000000001,"15min":21.931000000000001},"min":{"1min":11.138999999999999,"5min":10.387,"15min":9.5990000000000002},"max":{"1min":156.46199999999999,"5min":156.46199999999999,"15min":227.40899999999999},"last":19.469999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:12 2023","interfaces":[{"interface":"back","average":{"1min":26.216999999999999,"5min":17.390000000000001,"15min":20.870000000000001},"min":{"1min":10.401999999999999,"5min":10.08,"15min":9.6539999999999999},"max":{"1min":91.260000000000005,"5min":91.260000000000005,"15min":246.178},"last":16.042000000000002},{"interface":"front","average":{"1min":21.318999999999999,"5min":13.746,"15min":16.047999999999998},"min":{"1min":7.742,"5min":7.0510000000000002,"15min":7.0510000000000002},"max":{"1min":82.622,"5min":82.622,"15min":173.66800000000001},"last":11.932}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":40.828000000000003,"5min":19.824000000000002,"15min":23.663},"min":{"1min":10.592000000000001,"5min":7.5250000000000004,"15min":7.5250000000000004},"max":{"1min":119.569,"5min":119.569,"15min":246.55000000000001},"last":14.65},{"interface":"front","average":{"1min":49.973999999999997,"5min":16.795999999999999,"15min":17.120999999999999},"min":{"1min":6.2610000000000001,"5min":5.4429999999999996,"15min":5.4429999999999996},"max":{"1min":203.21799999999999,"5min":203.21799999999999,"15min":205.15799999999999},"last":6.2759999999999998}]},{"osd":6,"last update":"Mon Dec 11 20:48:12 2023","interfaces":[{"interface":"back","average":{"1min":30.149999999999999,"5min":19.039000000000001,"15min":25.846},"min":{"1min":11.76,"5min":10.831,"15min":9.9320000000000004},"max":{"1min":114.998,"5min":114.998,"15min":266.13499999999999},"last":17.385000000000002},{"interface":"front","average":{"1min":28.504000000000001,"5min":17.774999999999999,"15min":23.965},"min":{"1min":11.42,"5min":9.8339999999999996,"15min":8.391},"max":{"1min":115.377,"5min":115.377,"15min":208.00299999999999},"last":16.800000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":60.445,"5min":26.123000000000001,"15min":21.013999999999999},"min":{"1min":12.372,"5min":12.295999999999999,"15min":6.4690000000000003},"max":{"1min":260.45999999999998,"5min":260.45999999999998,"15min":260.45999999999998},"last":20.302},{"interface":"front","average":{"1min":44.287999999999997,"5min":22.015999999999998,"15min":25.881},"min":{"1min":12.273999999999999,"5min":10.863,"15min":10.863},"max":{"1min":172.559,"5min":172.559,"15min":264.99299999999999},"last":18.449999999999999}]}]},{"osd":2,"up_from":16,"seq":68719477122,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65216,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306624,"statfs":{"total":96636764160,"available":96569982976,"internally_reserved":0,"allocated":2048000,"data_stored":526593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":20.559000000000001,"5min":12.036,"15min":15.512},"min":{"1min":5.0780000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"max":{"1min":89.881,"5min":89.881,"15min":200.447},"last":18.100999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":11.49,"15min":16.253},"min":{"1min":5.6619999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":92.114000000000004,"5min":92.114000000000004,"15min":198.22900000000001},"last":19.876999999999999}]},{"osd":1,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":21.713000000000001,"5min":13.699,"15min":16.245999999999999},"min":{"1min":7.0590000000000002,"5min":5.7869999999999999,"15min":5.7869999999999999},"max":{"1min":89.183999999999997,"5min":89.183999999999997,"15min":143.56},"last":7.1269999999999998},{"interface":"front","average":{"1min":27.739999999999998,"5min":15.618,"15min":17.497},"min":{"1min":6.1879999999999997,"5min":6.1879999999999997,"15min":6.1879999999999997},"max":{"1min":175.16200000000001,"5min":175.16200000000001,"15min":175.16200000000001},"last":31.603000000000002}]},{"osd":3,"last update":"Mon Dec 11 20:48:04 2023","interfaces":[{"interface":"back","average":{"1min":8.5960000000000001,"5min":9.266,"15min":13.256},"min":{"1min":5.7960000000000003,"5min":5.3550000000000004,"15min":5.3550000000000004},"max":{"1min":15.034000000000001,"5min":21.725999999999999,"15min":171.71700000000001},"last":18.925000000000001},{"interface":"front","average":{"1min":13.776999999999999,"5min":13.532999999999999,"15min":17.481000000000002},"min":{"1min":9.9380000000000006,"5min":6.7169999999999996,"15min":5.9749999999999996},"max":{"1min":20.417000000000002,"5min":22.396000000000001,"15min":182.63399999999999},"last":24.898}]},{"osd":4,"last update":"Mon Dec 11 20:48:04 2023","interfaces":[{"interface":"back","average":{"1min":13.311999999999999,"5min":13.516,"15min":18.062999999999999},"min":{"1min":8.7010000000000005,"5min":7.6349999999999998,"15min":7.6349999999999998},"max":{"1min":18.588999999999999,"5min":27.902999999999999,"15min":161.84899999999999},"last":17.302},{"interface":"front","average":{"1min":16.510000000000002,"5min":17.042000000000002,"15min":22.538},"min":{"1min":10.923999999999999,"5min":10.923999999999999,"15min":9.9870000000000001},"max":{"1min":21.420999999999999,"5min":31.385999999999999,"15min":183.25899999999999},"last":21.986000000000001}]},{"osd":5,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":49.709000000000003,"5min":22.446999999999999,"15min":21.960999999999999},"min":{"1min":7.8159999999999998,"5min":7.8159999999999998,"15min":7.8159999999999998},"max":{"1min":364.11200000000002,"5min":364.11200000000002,"15min":364.11200000000002},"last":34.593000000000004},{"interface":"front","average":{"1min":39.561999999999998,"5min":20.593,"15min":24.591999999999999},"min":{"1min":10.625999999999999,"5min":9.1349999999999998,"15min":8.5020000000000007},"max":{"1min":296.74000000000001,"5min":296.74000000000001,"15min":296.74000000000001},"last":25.559000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:18 2023","interfaces":[{"interface":"back","average":{"1min":24.114999999999998,"5min":14.093999999999999,"15min":17.513999999999999},"min":{"1min":6.6799999999999997,"5min":6.6799999999999997,"15min":6.0990000000000002},"max":{"1min":251.19300000000001,"5min":251.19300000000001,"15min":251.19300000000001},"last":27.727},{"interface":"front","average":{"1min":17.216000000000001,"5min":16.463999999999999,"15min":21.641999999999999},"min":{"1min":12.948,"5min":9.4960000000000004,"15min":9.4469999999999992},"max":{"1min":30.542000000000002,"5min":30.542000000000002,"15min":181.96299999999999},"last":26.811}]},{"osd":7,"last update":"Mon Dec 11 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":17.181999999999999,"5min":18.201000000000001,"15min":23.617000000000001},"min":{"1min":10.379,"5min":10.379,"15min":10.379},"max":{"1min":22.295999999999999,"5min":30.533000000000001,"15min":197.27199999999999},"last":33.119},{"interface":"front","average":{"1min":13.143000000000001,"5min":13.51,"15min":18.437999999999999},"min":{"1min":8.0709999999999997,"5min":8.0199999999999996,"15min":7.085},"max":{"1min":18.725000000000001,"5min":23.817,"15min":233.57400000000001},"last":28.289000000000001}]}]},{"osd":1,"up_from":16,"seq":68719477123,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69784,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302056,"statfs":{"total":96636764160,"available":96565305344,"internally_reserved":0,"allocated":2461696,"data_stored":931905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":7.8470000000000004,"15min":13.646000000000001},"min":{"1min":4.6870000000000003,"5min":4.0949999999999998,"15min":3.903},"max":{"1min":72.888000000000005,"5min":72.888000000000005,"15min":283.91899999999998},"last":6.0129999999999999},{"interface":"front","average":{"1min":17.266999999999999,"5min":9.9909999999999997,"15min":14.855},"min":{"1min":5.8250000000000002,"5min":4.9340000000000002,"15min":4.6879999999999997},"max":{"1min":99.680000000000007,"5min":99.680000000000007,"15min":391.19999999999999},"last":20.667999999999999}]},{"osd":2,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":18.952999999999999,"5min":13.276,"15min":19.867999999999999},"min":{"1min":7.7510000000000003,"5min":6.0300000000000002,"15min":6.0300000000000002},"max":{"1min":84.167000000000002,"5min":84.167000000000002,"15min":389.66800000000001},"last":8.6199999999999992},{"interface":"front","average":{"1min":19.573,"5min":14.933999999999999,"15min":20.850000000000001},"min":{"1min":9.2989999999999995,"5min":9.2590000000000003,"15min":6.7670000000000003},"max":{"1min":100.836,"5min":100.836,"15min":376.82100000000003},"last":9.7769999999999992}]},{"osd":3,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":20.777999999999999,"5min":12.682,"15min":24.100999999999999},"min":{"1min":6.9080000000000004,"5min":5.7610000000000001,"15min":5.7610000000000001},"max":{"1min":101.74299999999999,"5min":101.74299999999999,"15min":421.89400000000001},"last":24.378},{"interface":"front","average":{"1min":29.260000000000002,"5min":16.670000000000002,"15min":20.762},"min":{"1min":10.404,"5min":5.7220000000000004,"15min":5.7220000000000004},"max":{"1min":175.047,"5min":175.047,"15min":293.06099999999998},"last":26.821000000000002}]},{"osd":4,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":12.462999999999999,"15min":18.878},"min":{"1min":6.6740000000000004,"5min":6.1630000000000003,"15min":5.952},"max":{"1min":120.863,"5min":120.863,"15min":342.28199999999998},"last":25.181999999999999},{"interface":"front","average":{"1min":33.405000000000001,"5min":18.756,"15min":23.010000000000002},"min":{"1min":12.538,"5min":10.499000000000001,"15min":6.7039999999999997},"max":{"1min":187.15100000000001,"5min":187.15100000000001,"15min":289.68799999999999},"last":28.654}]},{"osd":5,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":25.463999999999999,"5min":17.777000000000001,"15min":22.192},"min":{"1min":11.183,"5min":11.183,"15min":7.4859999999999998},"max":{"1min":183.84700000000001,"5min":183.84700000000001,"15min":556.32500000000005},"last":29.917000000000002},{"interface":"front","average":{"1min":27.771000000000001,"5min":19.635999999999999,"15min":24.861000000000001},"min":{"1min":12.422000000000001,"5min":12.422000000000001,"15min":9.5739999999999998},"max":{"1min":186.06700000000001,"5min":186.06700000000001,"15min":309.16399999999999},"last":33.709000000000003}]},{"osd":6,"last update":"Mon Dec 11 20:48:43 2023","interfaces":[{"interface":"back","average":{"1min":23.209,"5min":13.757,"15min":20.975000000000001},"min":{"1min":8.6910000000000007,"5min":7.5270000000000001,"15min":6.7590000000000003},"max":{"1min":169.429,"5min":169.429,"15min":372.90899999999999},"last":27.663},{"interface":"front","average":{"1min":29.114999999999998,"5min":19.785,"15min":27.733000000000001},"min":{"1min":12.670999999999999,"5min":12.670999999999999,"15min":11.641999999999999},"max":{"1min":197.87299999999999,"5min":197.87299999999999,"15min":297.71499999999997},"last":35.917999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":20.015000000000001,"5min":13.327,"15min":22.792999999999999},"min":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"max":{"1min":109.782,"5min":109.782,"15min":426.10700000000003},"last":32.804000000000002},{"interface":"front","average":{"1min":33.633000000000003,"5min":21.547000000000001,"15min":29.686},"min":{"1min":13.068,"5min":13.068,"15min":11.789},"max":{"1min":189.21199999999999,"5min":189.21199999999999,"15min":331.24799999999999},"last":44.869}]}]},{"osd":0,"up_from":16,"seq":68719477125,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65416,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306424,"statfs":{"total":96636764160,"available":96569778176,"internally_reserved":0,"allocated":2064384,"data_stored":538161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":11.036,"15min":16.446000000000002},"min":{"1min":4.609,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":98.932000000000002,"5min":98.932000000000002,"15min":192.989},"last":10.585000000000001},{"interface":"front","average":{"1min":20.565000000000001,"5min":11.683999999999999,"15min":17.998999999999999},"min":{"1min":6.1719999999999997,"5min":5.2220000000000004,"15min":4.7300000000000004},"max":{"1min":101.66,"5min":101.66,"15min":288.91800000000001},"last":6.6210000000000004}]},{"osd":2,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":22.863,"5min":13.515000000000001,"15min":17.582999999999998},"min":{"1min":7.694,"5min":7.0620000000000003,"15min":5.2190000000000003},"max":{"1min":96.489999999999995,"5min":96.489999999999995,"15min":294.733},"last":5.5389999999999997},{"interface":"front","average":{"1min":29.260999999999999,"5min":17.803999999999998,"15min":23.199999999999999},"min":{"1min":11.227,"5min":9.9109999999999996,"15min":7.8630000000000004},"max":{"1min":126.831,"5min":126.831,"15min":306.08100000000002},"last":9.7769999999999992}]},{"osd":3,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.068000000000001,"5min":13.037000000000001,"15min":15.56},"min":{"1min":6.1769999999999996,"5min":5.9829999999999997,"15min":5.5979999999999999},"max":{"1min":160.999,"5min":160.999,"15min":166.49100000000001},"last":22.376000000000001},{"interface":"front","average":{"1min":29.303000000000001,"5min":17.632999999999999,"15min":20.312999999999999},"min":{"1min":10.973000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"max":{"1min":177.35400000000001,"5min":177.35400000000001,"15min":177.35400000000001},"last":11.167}]},{"osd":4,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.286000000000001,"5min":16.172999999999998,"15min":19.902999999999999},"min":{"1min":6.5970000000000004,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":169.92599999999999,"5min":169.92599999999999,"15min":169.92599999999999},"last":26.155000000000001},{"interface":"front","average":{"1min":31.260999999999999,"5min":19.547000000000001,"15min":20.800000000000001},"min":{"1min":10.371,"5min":10.371,"15min":6.6859999999999999},"max":{"1min":149.65600000000001,"5min":149.65600000000001,"15min":153.42500000000001},"last":11.763999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.992999999999999,"5min":13.356,"15min":15.753},"min":{"1min":7.0940000000000003,"5min":6.5999999999999996,"15min":6.234},"max":{"1min":110.476,"5min":110.476,"15min":210.374},"last":23.175000000000001},{"interface":"front","average":{"1min":30.498999999999999,"5min":16.635999999999999,"15min":20.227},"min":{"1min":8.7940000000000005,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":110.836,"5min":110.836,"15min":267.03199999999998},"last":28.155000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":24.010999999999999,"5min":14.885,"15min":18.350999999999999},"min":{"1min":9.0850000000000009,"5min":8.0050000000000008,"15min":6.4790000000000001},"max":{"1min":111.423,"5min":111.423,"15min":161.09899999999999},"last":101.5},{"interface":"front","average":{"1min":33.442999999999998,"5min":19.559000000000001,"15min":23.120000000000001},"min":{"1min":11.138999999999999,"5min":10.305999999999999,"15min":9.4440000000000008},"max":{"1min":186.089,"5min":186.089,"15min":246.738},"last":113.024}]},{"osd":7,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.001000000000001,"5min":19.183,"15min":21.738},"min":{"1min":9.4049999999999994,"5min":9.4049999999999994,"15min":8.8789999999999996},"max":{"1min":187.00299999999999,"5min":187.00299999999999,"15min":187.00299999999999},"last":99.403999999999996},{"interface":"front","average":{"1min":29.199999999999999,"5min":19.706,"15min":23.911999999999999},"min":{"1min":12.224,"5min":11.782999999999999,"15min":7.7640000000000002},"max":{"1min":127.563,"5min":127.563,"15min":264.45100000000002},"last":13.661}]}]},{"osd":4,"up_from":14,"seq":60129542529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65168,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306672,"statfs":{"total":96636764160,"available":96570032128,"internally_reserved":0,"allocated":2093056,"data_stored":548539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":13.959,"5min":10.147,"15min":15.048999999999999},"min":{"1min":4.5049999999999999,"5min":4.4409999999999998,"15min":4.1879999999999997},"max":{"1min":40.877000000000002,"5min":40.877000000000002,"15min":248.19900000000001},"last":6.0069999999999997},{"interface":"front","average":{"1min":17.295999999999999,"5min":10.696999999999999,"15min":13.792},"min":{"1min":5.1589999999999998,"5min":5.1059999999999999,"15min":4.1449999999999996},"max":{"1min":77.028000000000006,"5min":77.028000000000006,"15min":169.541},"last":24.709}]},{"osd":1,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":23.628,"5min":14.050000000000001,"15min":18.832000000000001},"min":{"1min":7.1269999999999998,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":116.298,"5min":116.298,"15min":310.53500000000003},"last":7.1269999999999998},{"interface":"front","average":{"1min":21.18,"5min":13.920999999999999,"15min":18.367999999999999},"min":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":4.7380000000000004},"max":{"1min":109.753,"5min":109.753,"15min":269.65100000000001},"last":25.824000000000002}]},{"osd":2,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":14.571999999999999,"5min":13.747999999999999,"15min":18.155000000000001},"min":{"1min":8.3030000000000008,"5min":8.3030000000000008,"15min":5.0830000000000002},"max":{"1min":24.594000000000001,"5min":24.594000000000001,"15min":255.31700000000001},"last":9.1799999999999997},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.42,"15min":17.867000000000001},"min":{"1min":6.6429999999999998,"5min":6.6310000000000002,"15min":5.4199999999999999},"max":{"1min":16.919,"5min":25.324999999999999,"15min":291.67899999999997},"last":16.722999999999999}]},{"osd":3,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":13.087,"5min":12.433999999999999,"15min":18.640999999999998},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.2460000000000004},"max":{"1min":23.73,"5min":23.73,"15min":260.55099999999999},"last":17.395},{"interface":"front","average":{"1min":16.361000000000001,"5min":15.35,"15min":22.067},"min":{"1min":9.2949999999999999,"5min":8.1969999999999992,"15min":7.0030000000000001},"max":{"1min":20.364999999999998,"5min":21.821000000000002,"15min":320.53899999999999},"last":20.867000000000001}]},{"osd":5,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":26.222000000000001,"5min":18.422999999999998,"15min":21.838999999999999},"min":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":7.7359999999999998},"max":{"1min":132.37799999999999,"5min":132.37799999999999,"15min":298.69499999999999},"last":23.280999999999999},{"interface":"front","average":{"1min":22.783999999999999,"5min":15.782,"15min":21.024999999999999},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":81.689999999999998,"5min":81.689999999999998,"15min":309.88200000000001},"last":46.890999999999998}]},{"osd":6,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":16.067,"5min":14.606999999999999,"15min":18.742000000000001},"min":{"1min":10.874000000000001,"5min":7.6509999999999998,"15min":5.4459999999999997},"max":{"1min":30.559000000000001,"5min":30.559000000000001,"15min":290.57600000000002},"last":26.710999999999999},{"interface":"front","average":{"1min":12.907999999999999,"5min":14.144,"15min":18.532},"min":{"1min":7.3540000000000001,"5min":7.3540000000000001,"15min":6.5389999999999997},"max":{"1min":27.225999999999999,"5min":27.225999999999999,"15min":285.52499999999998},"last":28.818000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":21.823,"5min":17.431000000000001,"15min":21.483000000000001},"min":{"1min":10.587,"5min":10.176,"15min":7.2599999999999998},"max":{"1min":69.731999999999999,"5min":69.731999999999999,"15min":297.63299999999998},"last":20.187999999999999},{"interface":"front","average":{"1min":23.029,"5min":16.969999999999999,"15min":23.030000000000001},"min":{"1min":10.496,"5min":7.3639999999999999,"15min":7.3639999999999999},"max":{"1min":98.728999999999999,"5min":98.728999999999999,"15min":364.18799999999999},"last":47.399000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575236,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65128,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306712,"statfs":{"total":96636764160,"available":96570073088,"internally_reserved":0,"allocated":2060288,"data_stored":534065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:16 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":10.074999999999999,"15min":17.242999999999999},"min":{"1min":4.3600000000000003,"5min":4.3600000000000003,"15min":3.8969999999999998},"max":{"1min":15.532999999999999,"5min":16.571000000000002,"15min":280.62200000000001},"last":10.353},{"interface":"front","average":{"1min":9.0850000000000009,"5min":9.4939999999999998,"15min":17.504000000000001},"min":{"1min":6.3550000000000004,"5min":5.5990000000000002,"15min":4.9800000000000004},"max":{"1min":12.73,"5min":24.454999999999998,"15min":294.44299999999998},"last":6.101}]},{"osd":1,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":11.183999999999999,"5min":11.531000000000001,"15min":16.939},"min":{"1min":6.4340000000000002,"5min":5.5519999999999996,"15min":5.4489999999999998},"max":{"1min":18.149999999999999,"5min":18.594000000000001,"15min":217.40100000000001},"last":8.9510000000000005},{"interface":"front","average":{"1min":12.154,"5min":12.77,"15min":18.738},"min":{"1min":8.1039999999999992,"5min":6.3730000000000002,"15min":6.3730000000000002},"max":{"1min":16.456,"5min":18.507999999999999,"15min":253.15799999999999},"last":14.775}]},{"osd":2,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":12.81,"5min":13.550000000000001,"15min":18.916},"min":{"1min":7.8259999999999996,"5min":7.7839999999999998,"15min":6.319},"max":{"1min":16.373000000000001,"5min":20.346,"15min":275.31},"last":17.143000000000001},{"interface":"front","average":{"1min":15.932,"5min":16.280000000000001,"15min":24.547999999999998},"min":{"1min":11.714,"5min":11.714,"15min":11.016},"max":{"1min":19.315999999999999,"5min":22.353000000000002,"15min":399.96100000000001},"last":17.960000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":8.4879999999999995,"5min":8.3040000000000003,"15min":13.114000000000001},"min":{"1min":5.1879999999999997,"5min":5.1879999999999997,"15min":5.1879999999999997},"max":{"1min":13.698,"5min":16.690999999999999,"15min":239.00200000000001},"last":7.0570000000000004},{"interface":"front","average":{"1min":14.753,"5min":15.654999999999999,"15min":21.408000000000001},"min":{"1min":10.894,"5min":9.6389999999999993,"15min":9.2240000000000002},"max":{"1min":17.164000000000001,"5min":21.308,"15min":263.565},"last":13.917999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":27.109999999999999,"5min":15.597,"15min":16.378},"min":{"1min":9.6489999999999991,"5min":6.516,"15min":6.1900000000000004},"max":{"1min":166.67099999999999,"5min":166.67099999999999,"15min":201.459},"last":12.923999999999999},{"interface":"front","average":{"1min":22.686,"5min":13.342000000000001,"15min":17.608000000000001},"min":{"1min":7.4580000000000002,"5min":6.2510000000000003,"15min":6.2510000000000003},"max":{"1min":124.88,"5min":124.88,"15min":203.36600000000001},"last":9.9649999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":39.771000000000001,"5min":21.721,"15min":26.289000000000001},"min":{"1min":12.252000000000001,"5min":10.476000000000001,"15min":9.7810000000000006},"max":{"1min":206.904,"5min":206.904,"15min":268.98000000000002},"last":15.65},{"interface":"front","average":{"1min":28.629000000000001,"5min":16.504999999999999,"15min":19.289999999999999},"min":{"1min":8.9410000000000007,"5min":8.2639999999999993,"15min":7.6360000000000001},"max":{"1min":127.8,"5min":127.8,"15min":283.61900000000003},"last":11.708}]},{"osd":6,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":22.581,"5min":16.634,"15min":21.099},"min":{"1min":12.592000000000001,"5min":8.8339999999999996,"15min":6.6479999999999997},"max":{"1min":115.38800000000001,"5min":115.38800000000001,"15min":282.71199999999999},"last":11.077999999999999},{"interface":"front","average":{"1min":25.696999999999999,"5min":19.786000000000001,"15min":25.102},"min":{"1min":15.474,"5min":10.999000000000001,"15min":9.7569999999999997},"max":{"1min":126.97199999999999,"5min":126.97199999999999,"15min":260.13400000000001},"last":16.382000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575234,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69696,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302144,"statfs":{"total":96636764160,"available":96565395456,"internally_reserved":0,"allocated":2478080,"data_stored":939549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":13.065,"5min":10.132999999999999,"15min":13.657999999999999},"min":{"1min":4.4960000000000004,"5min":4.4509999999999996,"15min":4.3849999999999998},"max":{"1min":52.319000000000003,"5min":52.319000000000003,"15min":121.80500000000001},"last":18.257000000000001},{"interface":"front","average":{"1min":13.058999999999999,"5min":9.7390000000000008,"15min":16.152999999999999},"min":{"1min":6.3780000000000001,"5min":4.5369999999999999,"15min":4.4349999999999996},"max":{"1min":49.723999999999997,"5min":49.723999999999997,"15min":307.66000000000003},"last":6.7350000000000003}]},{"osd":1,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":12.073,"5min":9.8230000000000004,"15min":13.728},"min":{"1min":6.1070000000000002,"5min":5.3449999999999998,"15min":4.5490000000000004},"max":{"1min":57.738999999999997,"5min":57.738999999999997,"15min":131.70500000000001},"last":118.562},{"interface":"front","average":{"1min":14.811,"5min":11.605,"15min":16.902000000000001},"min":{"1min":7.4320000000000004,"5min":6.0609999999999999,"15min":5.7169999999999996},"max":{"1min":60.664999999999999,"5min":60.664999999999999,"15min":202.07300000000001},"last":24.960000000000001}]},{"osd":2,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.559000000000001,"5min":15.26,"15min":18.596},"min":{"1min":8.6720000000000006,"5min":7.7460000000000004,"15min":6.9470000000000001},"max":{"1min":109.023,"5min":109.023,"15min":203.67699999999999},"last":23.725000000000001},{"interface":"front","average":{"1min":23.201000000000001,"5min":15.288,"15min":19.117000000000001},"min":{"1min":9.0129999999999999,"5min":7.8760000000000003,"15min":6.8719999999999999},"max":{"1min":93.819999999999993,"5min":93.819999999999993,"15min":211.67500000000001},"last":46.896000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":15.050000000000001,"15min":19.510000000000002},"min":{"1min":8.3170000000000002,"5min":8.0700000000000003,"15min":7.548},"max":{"1min":58.795000000000002,"5min":58.795000000000002,"15min":167.095},"last":22.346},{"interface":"front","average":{"1min":19.649000000000001,"5min":15.371,"15min":19.989000000000001},"min":{"1min":10.052,"5min":8.1690000000000005,"15min":7.6769999999999996},"max":{"1min":68.209999999999994,"5min":68.209999999999994,"15min":207.58799999999999},"last":10.401999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.547000000000001,"5min":16.661000000000001,"15min":19.132000000000001},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":6.8360000000000003},"max":{"1min":69.195999999999998,"5min":69.195999999999998,"15min":132.64500000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":33.280999999999999,"5min":20.861000000000001,"15min":23.035},"min":{"1min":14.875999999999999,"5min":9.9390000000000001,"15min":8.2590000000000003},"max":{"1min":199.828,"5min":199.828,"15min":199.828},"last":20.135999999999999}]},{"osd":6,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":23.556999999999999,"5min":12.887,"15min":16.77},"min":{"1min":6.3620000000000001,"5min":6.234,"15min":5.8789999999999996},"max":{"1min":199.03800000000001,"5min":199.03800000000001,"15min":199.03800000000001},"last":119.57299999999999},{"interface":"front","average":{"1min":26.257999999999999,"5min":19.350999999999999,"15min":19.757000000000001},"min":{"1min":11.548999999999999,"5min":11.548999999999999,"15min":6.1479999999999997},"max":{"1min":113.911,"5min":113.911,"15min":268.072},"last":111.997}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":22.385000000000002,"5min":19.629999999999999,"15min":22.059999999999999},"min":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":8.3239999999999998},"max":{"1min":63.686999999999998,"5min":63.686999999999998,"15min":198.304},"last":20.847000000000001},{"interface":"front","average":{"1min":19.053000000000001,"5min":15.726000000000001,"15min":20.466000000000001},"min":{"1min":10.156000000000001,"5min":8.0630000000000006,"15min":8.0630000000000006},"max":{"1min":56.078000000000003,"5min":56.078000000000003,"15min":181.84299999999999},"last":7.7069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-11T20:49:04.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-11T20:49:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.360+0000 7f5b7319e640 10 monclient: get_monmap_and_config 2023-12-11T20:49:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.360+0000 7f5b7319e640 10 monclient: build_initial_monmap 2023-12-11T20:49:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.360+0000 7f5b7319e640 10 monclient: monmap: 2023-12-11T20:49:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:04.361691+0000 2023-12-11T20:49:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:04.361691+0000 2023-12-11T20:49:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:04.362 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.360+0000 7f5b7319e640 1 Processor -- start 2023-12-11T20:49:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.360+0000 7f5b7319e640 1 -- start start 2023-12-11T20:49:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.360+0000 7f5b7319e640 10 monclient: init 2023-12-11T20:49:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-12-11T20:49:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5b6c0a8a20 0x7f5b6c13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient(hunting): picked mon.noname-c con 0x7f5b6c0a8a20 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8e00 0x7f5b6c13f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient(hunting): picked mon.noname-b con 0x7f5b6c0a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b6c1423b0 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c0b21f0 con 0x7f5b6c1423b0 2023-12-11T20:49:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c0a9cc0 con 0x7f5b6c0a8a20 2023-12-11T20:49:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c142f50 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.361+0000 7f5b7319e640 10 monclient(hunting): authenticate will time out at 3430.691668s 2023-12-11T20:49:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8e00 0x7f5b6c13f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b70f3a640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5b6c0a8a20 0x7f5b6c13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8e00 0x7f5b6c13f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39290/0 (socket says 172.21.15.86:39290) 2023-12-11T20:49:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 1 -- 172.21.15.86:0/1382857348 learned_addr learned my addr 172.21.15.86:0/1382857348 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 10 monclient(hunting): get_auth_request con 0x7f5b6c0a8e00 auth_method 0 2023-12-11T20:49:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b70f3a640 10 monclient(hunting): get_auth_request con 0x7f5b6c0a8a20 auth_method 0 2023-12-11T20:49:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b70f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b70f3a640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b70f3a640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b70f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b70f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.362+0000 7f5b70f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 10 monclient(hunting): handle_auth_done global_id 21013 payload 274 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 1 -- 172.21.15.86:0/1382857348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8e00 msgr2=0x7f5b6c13f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 1 --2- 172.21.15.86:0/1382857348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8e00 0x7f5b6c13f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 1 -- 172.21.15.86:0/1382857348 >> v1:172.21.15.133:6789/0 conn(0x7f5b6c1423b0 legacy=0x7f5b6c142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 10 monclient: _finish_hunting 0 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 1 monclient: found mon.noname-c 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:49:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b70f3a640 1 -- 172.21.15.86:0/1382857348 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c13f930 con 0x7f5b6c0a8a20 2023-12-11T20:49:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.364+0000 7f5b6bfff640 1 --2- 172.21.15.86:0/1382857348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8e00 0x7f5b6c13f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:49:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.367+0000 7f5b70f3a640 1 --2- 172.21.15.86:0/1382857348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8a20 0x7f5b6c13eeb0 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 crypto rx=0x7f5b600097a0 tx=0x7f5b6002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9efce3995d954f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 1 -- 172.21.15.86:0/1382857348 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60030710 con 0x7f5b6c0a8a20 2023-12-11T20:49:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: dump: 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 1 -- 172.21.15.86:0/1382857348 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6002e050 con 0x7f5b6c0a8a20 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 1 -- 172.21.15.86:0/1382857348 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60030a90 con 0x7f5b6c0a8a20 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 5 monclient: authenticate success, global_id 21013 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 4 set_mon_vals no callback set 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b6b7fe640 10 monclient: dump: 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 10 monclient: shutdown 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 1 -- 172.21.15.86:0/1382857348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8a20 msgr2=0x7f5b6c13eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 1 --2- 172.21.15.86:0/1382857348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8a20 0x7f5b6c13eeb0 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 crypto rx=0x7f5b600097a0 tx=0x7f5b6002b710 comp rx=0 tx=0).stop 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 1 -- 172.21.15.86:0/1382857348 shutdown_connections 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 1 --2- 172.21.15.86:0/1382857348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8e00 0x7f5b6c13f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 1 --2- 172.21.15.86:0/1382857348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8a20 0x7f5b6c13eeb0 unknown :-1 s=CLOSED pgs=5108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.371+0000 7f5b7319e640 1 -- 172.21.15.86:0/1382857348 >> 172.21.15.86:0/1382857348 conn(0x7f5b6c0b7870 msgr2=0x7f5b6c0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.372+0000 7f5b7319e640 1 -- 172.21.15.86:0/1382857348 shutdown_connections 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.372+0000 7f5b7319e640 1 -- 172.21.15.86:0/1382857348 wait complete. 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.372+0000 7f5b7319e640 10 monclient: build_initial_monmap 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.372+0000 7f5b7319e640 10 monclient: monmap: 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.372+0000 7f5b7319e640 1 librados: starting msgr at 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.372+0000 7f5b7319e640 1 librados: starting objecter 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.372+0000 7f5b7319e640 1 Processor -- start 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 -- start start 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 librados: setting wanted keys 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 librados: calling monclient init 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient: init 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8a20 0x7f5b6c137f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient(hunting): picked mon.noname-c con 0x7f5b6c0a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8e00 0x7f5b6c143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient(hunting): picked mon.noname-b con 0x7f5b6c0a8e00 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b6c1423b0 0x7f5b6c144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient(hunting): picked mon.noname-a con 0x7f5b6c1423b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c139700 con 0x7f5b6c1423b0 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c139970 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c145590 con 0x7f5b6c0a8a20 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.373+0000 7f5b7319e640 10 monclient(hunting): authenticate will time out at 3430.703858s 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8a20 0x7f5b6c137f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b7173b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b6c1423b0 0x7f5b6c144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b6bfff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8e00 0x7f5b6c143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8a20 0x7f5b6c137f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39306/0 (socket says 172.21.15.86:39306) 2023-12-11T20:49:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 1 -- 172.21.15.86:0/3642286745 learned_addr learned my addr 172.21.15.86:0/3642286745 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 10 monclient(hunting): get_auth_request con 0x7f5b6c0a8a20 auth_method 0 2023-12-11T20:49:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b6bfff640 10 monclient(hunting): get_auth_request con 0x7f5b6c0a8e00 auth_method 0 2023-12-11T20:49:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b6bfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b6bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b7173b640 10 monclient(hunting): get_auth_request con 0x7f5b6c1423b0 auth_method 0 2023-12-11T20:49:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b7173b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b7173b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b7173b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.374+0000 7f5b70f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.375+0000 7f5b6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.375+0000 7f5b6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.375+0000 7f5b6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.375+0000 7f5b7173b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.375+0000 7f5b7173b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.375+0000 7f5b7173b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 10 monclient(hunting): handle_auth_done global_id 21016 payload 824 2023-12-11T20:49:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 1 -- 172.21.15.86:0/3642286745 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8a20 msgr2=0x7f5b6c137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8a20 0x7f5b6c137f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 1 -- 172.21.15.86:0/3642286745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b6c1423b0 msgr2=0x7f5b6c144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b6c1423b0 0x7f5b6c144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 10 monclient: _finish_hunting 0 2023-12-11T20:49:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 1 monclient: found mon.noname-b 2023-12-11T20:49:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.377+0000 7f5b6bfff640 1 -- 172.21.15.86:0/3642286745 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b580094d0 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.379+0000 7f5b6bfff640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8e00 0x7f5b6c143390 secure :-1 s=READY pgs=5109 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800e980 tx=0x7f5b5800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.379+0000 7f5b697fa640 10 client.?.objecter ms_handle_connect 0x7f5b6c0a8e00 2023-12-11T20:49:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.379+0000 7f5b697fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 1 -- 172.21.15.86:0/3642286745 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5800ca00 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: dump: 2023-12-11T20:49:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:04.393 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:04.393 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:04.393 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:04.393 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:04.393 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:04.393 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: _finish_auth 0 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: _check_auth_tickets 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 1 -- 172.21.15.86:0/3642286745 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5800cb60 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 1 -- 172.21.15.86:0/3642286745 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58013470 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b697fa640 10 monclient: dump: 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 5 monclient: authenticate success, global_id 21016 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 10 monclient: _renew_subs 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b60009400 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b6affd640 4 set_mon_vals no callback set 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 10 client.21016.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 10 monclient: _renew_subs 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c145340 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.383+0000 7f5b7319e640 1 librados: init done 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.384+0000 7f5b7319e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.384+0000 7f5b7319e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.384+0000 7f5b7319e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:04.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.384+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c04e840 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.402+0000 7f5b697fa640 1 -- 172.21.15.86:0/3642286745 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5b580135d0 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.402+0000 7f5b697fa640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5b5006c850 0x7f5b5006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:04.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.402+0000 7f5b70f3a640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5b5006c850 0x7f5b5006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:04.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.403+0000 7f5b70f3a640 10 monclient: get_auth_request con 0x7f5b5006c850 auth_method 0 2023-12-11T20:49:04.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.403+0000 7f5b70f3a640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5b5006c850 0x7f5b5006ed10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5b6002b750 tx=0x7f5b6002f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:04.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.403+0000 7f5b697fa640 10 client.21016.objecter ms_handle_connect 0x7f5b5006c850 2023-12-11T20:49:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.628+0000 7f5b697fa640 1 -- 172.21.15.86:0/3642286745 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f5b58058d60 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.628+0000 7f5b697fa640 10 client.21016.objecter ms_dispatch 0x7f5b6c000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.628+0000 7f5b697fa640 3 client.21016.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.628+0000 7f5b697fa640 3 client.21016.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.628+0000 7f5b697fa640 20 client.21016.objecter dump_active .. 0 homeless 2023-12-11T20:49:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.639+0000 7f5b697fa640 1 -- 172.21.15.86:0/3642286745 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b580550a0 con 0x7f5b6c0a8e00 2023-12-11T20:49:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.639+0000 7f5b697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.639+0000 7f5b697fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:04.706 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:17.814 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.710 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:17.818 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.730 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:17.838 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.755 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:17.823 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.762 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:17.830 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.774 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:17.882 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.779 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:17.887 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.781 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:17.849 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.799 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:17.907 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:04.881 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.882+0000 7f5b7319e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:49:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.882+0000 7f5b7319e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:49:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.882+0000 7f5b7319e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:04.882+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b6c139ab0 con 0x7f5b6c0a8e00 2023-12-11T20:49:05.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.007+0000 7f5b697fa640 1 -- 172.21.15.86:0/3642286745 <== mon.1 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1057) v1 ==== 74+0+17863 (secure 0 0 0) 0x7f5b6c139ab0 con 0x7f5b6c0a8e00 2023-12-11T20:49:05.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.007+0000 7f5b697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-11T20:49:05.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.007+0000 7f5b697fa640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:49:05.011 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:49:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1057,"fsid":"bfc699ed-5134-41ce-a2e4-de833b93556d","created":"2023-12-11T20:15:58.839760+0000","modified":"2023-12-11T20:49:01.819614+0000","last_up_change":"2023-12-11T20:16:47.001015+0000","last_in_change":"2023-12-11T20:16:04.659077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":105,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-11T20:16:44.927047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-11T20:16:54.037205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-12-11T20:17:56.373176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"feaf0ee1-5c9d-4980-bb27-27c376912955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1053,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6821","nonce":1914311689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6826","nonce":1914311689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6831","nonce":1914311689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1914311689},{"type":"v1","addr":"172.21.15.86:6829","nonce":1914311689}]},"public_addr":"172.21.15.86:6821/1914311689","cluster_addr":"172.21.15.86:6826/1914311689","heartbeat_back_addr":"172.21.15.86:6831/1914311689","heartbeat_front_addr":"172.21.15.86:6829/1914311689","state":["exists","up"]},{"osd":1,"uuid":"f605c3b7-5dc2-4b26-a95e-eff31a2ab106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1045,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6809","nonce":4202858331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6811","nonce":4202858331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6815","nonce":4202858331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":4202858331},{"type":"v1","addr":"172.21.15.86:6813","nonce":4202858331}]},"public_addr":"172.21.15.86:6809/4202858331","cluster_addr":"172.21.15.86:6811/4202858331","heartbeat_back_addr":"172.21.15.86:6815/4202858331","heartbeat_front_addr":"172.21.15.86:6813/4202858331","state":["exists","up"]},{"osd":2,"uuid":"94937f44-381b-4c32-b869-184ceeb9ced0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1053,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6817","nonce":406445159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6819","nonce":406445159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6825","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6827","nonce":406445159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":406445159},{"type":"v1","addr":"172.21.15.86:6823","nonce":406445159}]},"public_addr":"172.21.15.86:6817/406445159","cluster_addr":"172.21.15.86:6819/406445159","heartbeat_back_addr":"172.21.15.86:6827/406445159","heartbeat_front_addr":"172.21.15.86:6823/406445159","state":["exists","up"]},{"osd":3,"uuid":"73d8e78f-aa7e-47e7-9ce1-5571f6fc0759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1045,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6801","nonce":4064445038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6803","nonce":4064445038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6807","nonce":4064445038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":4064445038},{"type":"v1","addr":"172.21.15.86:6805","nonce":4064445038}]},"public_addr":"172.21.15.86:6801/4064445038","cluster_addr":"172.21.15.86:6803/4064445038","heartbeat_back_addr":"172.21.15.86:6807/4064445038","heartbeat_front_addr":"172.21.15.86:6805/4064445038","state":["exists","up"]},{"osd":4,"uuid":"f9926b1b-1b80-43ad-beaa-cd6645b9b72c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1053,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6819","nonce":1433273204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6821","nonce":1433273204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6825","nonce":1433273204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1433273204},{"type":"v1","addr":"172.21.15.133:6823","nonce":1433273204}]},"public_addr":"172.21.15.133:6819/1433273204","cluster_addr":"172.21.15.133:6821/1433273204","heartbeat_back_addr":"172.21.15.133:6825/1433273204","heartbeat_front_addr":"172.21.15.133:6823/1433273204","state":["exists","up"]},{"osd":5,"uuid":"15e64f16-9d47-453f-a088-8c124c32c5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6803","nonce":3582621102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6805","nonce":3582621102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6809","nonce":3582621102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3582621102},{"type":"v1","addr":"172.21.15.133:6807","nonce":3582621102}]},"public_addr":"172.21.15.133:6803/3582621102","cluster_addr":"172.21.15.133:6805/3582621102","heartbeat_back_addr":"172.21.15.133:6809/3582621102","heartbeat_front_addr":"172.21.15.133:6807/3582621102","state":["exists","up"]},{"osd":6,"uuid":"b44f36ee-4393-49e2-a7cf-8f4cec897bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1053,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6827","nonce":1611390427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6829","nonce":1611390427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6833","nonce":1611390427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1611390427},{"type":"v1","addr":"172.21.15.133:6831","nonce":1611390427}]},"public_addr":"172.21.15.133:6827/1611390427","cluster_addr":"172.21.15.133:6829/1611390427","heartbeat_back_addr":"172.21.15.133:6833/1611390427","heartbeat_front_addr":"172.21.15.133:6831/1611390427","state":["exists","up"]},{"osd":7,"uuid":"5830a12f-1f5b-4e09-87a1-0bb9ae893db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1053,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6811","nonce":1136721322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6813","nonce":1136721322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6817","nonce":1136721322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1136721322},{"type":"v1","addr":"172.21.15.133:6815","nonce":1136721322}]},"public_addr":"172.21.15.133:6811/1136721322","cluster_addr":"172.21.15.133:6813/1136721322","heartbeat_back_addr":"172.21.15.133:6817/1136721322","heartbeat_front_addr":"172.21.15.133:6815/1136721322","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.469087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.889798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.985199+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.354101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:43.245493+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:41.942057+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:44.009608+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-11T20:16:42.326615+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":13,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":15,"snaps":[{"begin":9,"length":2}]},{"pool":34,"snaps":[{"begin":1,"length":1}]},{"pool":56,"snaps":[{"begin":1,"length":1}]},{"pool":76,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":11,"length":2}]},{"pool":87,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":100,"snaps":[{"begin":1,"length":1}]},{"pool":114,"snaps":[{"begin":1,"length":2}]},{"pool":118,"snaps":[{"begin":6,"length":2}]},{"pool":122,"snaps":[{"begin":3,"length":1}]},{"pool":126,"snaps":[{"begin":2,"length":2}]},{"pool":133,"snaps":[{"begin":2,"length":3}]},{"pool":139,"snaps":[{"begin":2,"length":1}]},{"pool":145,"snaps":[{"begin":2,"length":2}]},{"pool":149,"snaps":[{"begin":4,"length":4}]},{"pool":158,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":169,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":232,"snaps":[{"begin":12,"length":1}]},{"pool":253,"snaps":[{"begin":14,"length":1}]},{"pool":259,"snaps":[{"begin":17,"length":1}]},{"pool":260,"snaps":[{"begin":19,"length":2}]},{"pool":263,"snaps":[{"begin":24,"length":1}]},{"pool":306,"snaps":[{"begin":2,"length":1}]},{"pool":307,"snaps":[{"begin":4,"length":1}]},{"pool":310,"snaps":[{"begin":6,"length":1}]},{"pool":314,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-11T20:49:05.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 10 librados: watch_flush enter 2023-12-11T20:49:05.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 10 librados: watch_flush exit 2023-12-11T20:49:05.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 20 client.21016.objecter shutdown clearing up homeless session... 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 10 client.21016.objecter successfully canceled tick 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5b5006c850 msgr2=0x7f5b5006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5b5006c850 0x7f5b5006ed10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5b6002b750 tx=0x7f5b6002f000 comp rx=0 tx=0).stop 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 10 monclient: shutdown 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8e00 msgr2=0x7f5b6c143390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8e00 0x7f5b6c143390 secure :-1 s=READY pgs=5109 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800e980 tx=0x7f5b5800ee40 comp rx=0 tx=0).stop 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 shutdown_connections 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5b5006c850 0x7f5b5006ed10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b6c1423b0 0x7f5b6c144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0a8e00 0x7f5b6c143390 unknown :-1 s=CLOSED pgs=5109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 --2- 172.21.15.86:0/3642286745 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b6c0a8a20 0x7f5b6c137f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 >> 172.21.15.86:0/3642286745 conn(0x7f5b6c0b7870 msgr2=0x7f5b6c134670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:05.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 shutdown_connections 2023-12-11T20:49:05.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 -- 172.21.15.86:0/3642286745 wait complete. 2023-12-11T20:49:05.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:05.009+0000 7f5b7319e640 1 librados: shutdown 2023-12-11T20:49:06.022 INFO:tasks.ceph:Scrubbing osd.0 2023-12-11T20:49:06.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-11T20:49:06.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.130+0000 7f9b1451d640 10 monclient: get_monmap_and_config 2023-12-11T20:49:06.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.130+0000 7f9b1451d640 10 monclient: build_initial_monmap 2023-12-11T20:49:06.130 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.131+0000 7f9b1451d640 10 monclient: monmap: 2023-12-11T20:49:06.130 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:06.132491+0000 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:06.132491+0000 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:06.131 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.131+0000 7f9b1451d640 1 Processor -- start 2023-12-11T20:49:06.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.131+0000 7f9b1451d640 1 -- start start 2023-12-11T20:49:06.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.131+0000 7f9b1451d640 10 monclient: init 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b0c0af380 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f9b0c06fb40 0x7f9b0c0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b0c06fb40 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b0c13b4c0 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c0702b0 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c070590 con 0x7f9b0c0af380 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c13c020 con 0x7f9b0c06fb40 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:06.134 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.132+0000 7f9b1451d640 10 monclient(hunting): authenticate will time out at 3432.462652s 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.133+0000 7f9b12aba640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9b0c13b4c0 0x7f9b0c13b910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41806/0 (socket says 172.21.15.86:41806) 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.133+0000 7f9b12aba640 1 -- 172.21.15.86:0/2848971197 learned_addr learned my addr 172.21.15.86:0/2848971197 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.133+0000 7f9b11ab8640 1 --2- 172.21.15.86:0/2848971197 >> v2:172.21.15.133:3300/0 conn(0x7f9b0c06fb40 0x7f9b0c0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.133+0000 7f9b11ab8640 10 monclient(hunting): get_auth_request con 0x7f9b0c06fb40 auth_method 0 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.133+0000 7f9b11ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.133+0000 7f9b11ab8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.133+0000 7f9b11ab8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.134+0000 7f9b11ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.134+0000 7f9b11ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.134+0000 7f9b11ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b112b7640 1 -- 172.21.15.86:0/2848971197 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779263188 0 0) 0x7f9b0c0702b0 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b112b7640 10 monclient(hunting): _negotiate 2023-12-11T20:49:06.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b112b7640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b112b7640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b112b7640 10 monclient(hunting): my global_id is 16872 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b112b7640 1 -- 172.21.15.86:0/2848971197 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af8003520 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b11ab8640 10 monclient(hunting): handle_auth_done global_id 21019 payload 274 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b11ab8640 1 -- 172.21.15.86:0/2848971197 >> v1:172.21.15.133:6789/0 conn(0x7f9b0c0af380 legacy=0x7f9b0c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b11ab8640 1 -- 172.21.15.86:0/2848971197 >> v1:172.21.15.86:6789/0 conn(0x7f9b0c13b4c0 legacy=0x7f9b0c13b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b11ab8640 10 monclient: _finish_hunting 0 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b11ab8640 1 monclient: found mon.noname-c 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b11ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b11ab8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.135+0000 7f9b11ab8640 1 -- 172.21.15.86:0/2848971197 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c0add10 con 0x7f9b0c06fb40 2023-12-11T20:49:06.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.138+0000 7f9b11ab8640 1 --2- 172.21.15.86:0/2848971197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c06fb40 0x7f9b0c0727a0 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00a940 tx=0x7f9afc02cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22405712d3284947 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 1 -- 172.21.15.86:0/2848971197 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc02da20 con 0x7f9b0c06fb40 2023-12-11T20:49:06.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:06.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: dump: 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: _finish_auth 0 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: _check_auth_tickets 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:06.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 1 -- 172.21.15.86:0/2848971197 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc02db80 con 0x7f9b0c06fb40 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 5 monclient: authenticate success, global_id 21019 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 1 -- 172.21.15.86:0/2848971197 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc035440 con 0x7f9b0c06fb40 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 4 set_mon_vals no callback set 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b112b7640 10 monclient: dump: 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.144 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.144 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.144 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 10 monclient: shutdown 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 1 -- 172.21.15.86:0/2848971197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c06fb40 msgr2=0x7f9b0c0727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 1 --2- 172.21.15.86:0/2848971197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c06fb40 0x7f9b0c0727a0 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00a940 tx=0x7f9afc02cc70 comp rx=0 tx=0).stop 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 1 -- 172.21.15.86:0/2848971197 shutdown_connections 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 1 --2- 172.21.15.86:0/2848971197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c06fb40 0x7f9b0c0727a0 unknown :-1 s=CLOSED pgs=5110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 1 -- 172.21.15.86:0/2848971197 >> 172.21.15.86:0/2848971197 conn(0x7f9b0c0b3ef0 msgr2=0x7f9b0c0b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 1 -- 172.21.15.86:0/2848971197 shutdown_connections 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.142+0000 7f9b1451d640 1 -- 172.21.15.86:0/2848971197 wait complete. 2023-12-11T20:49:06.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 10 monclient: build_initial_monmap 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 10 monclient: monmap: 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 1 librados: starting msgr at 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 1 librados: starting objecter 2023-12-11T20:49:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 1 Processor -- start 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 1 -- start start 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 1 librados: setting wanted keys 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 1 librados: calling monclient init 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.143+0000 7f9b1451d640 10 monclient: init 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c06fb40 0x7f9b0c139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b0c06fb40 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c0af380 0x7f9b0c1d4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b0c0af380 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c13b4c0 0x7f9b0c1d5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b0c13b4c0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:06.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c132c80 con 0x7f9b0c06fb40 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c137720 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c137c20 con 0x7f9b0c0af380 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.144+0000 7f9b1451d640 10 monclient(hunting): authenticate will time out at 3432.474618s 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b122b9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c06fb40 0x7f9b0c139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b11ab8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c0af380 0x7f9b0c1d4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b12aba640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c13b4c0 0x7f9b0c1d5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b11ab8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c0af380 0x7f9b0c1d4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39320/0 (socket says 172.21.15.86:39320) 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b11ab8640 1 -- 172.21.15.86:0/57974444 learned_addr learned my addr 172.21.15.86:0/57974444 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b11ab8640 10 monclient(hunting): get_auth_request con 0x7f9b0c0af380 auth_method 0 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b11ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b11ab8640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b11ab8640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b12aba640 10 monclient(hunting): get_auth_request con 0x7f9b0c13b4c0 auth_method 0 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b12aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b12aba640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b12aba640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b122b9640 10 monclient(hunting): get_auth_request con 0x7f9b0c06fb40 auth_method 0 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b122b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b122b9640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b122b9640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b12aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b12aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b12aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b122b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b122b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.145+0000 7f9b122b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.146+0000 7f9b11ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.146+0000 7f9b11ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.146+0000 7f9b11ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 10 monclient(hunting): handle_auth_done global_id 21022 payload 824 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 1 -- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c0af380 msgr2=0x7f9b0c1d4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c0af380 0x7f9b0c1d4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 1 -- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c06fb40 msgr2=0x7f9b0c139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c06fb40 0x7f9b0c139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 10 monclient: _finish_hunting 0 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 1 monclient: found mon.noname-b 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.148+0000 7f9b12aba640 1 -- 172.21.15.86:0/57974444 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc00a5a0 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.150+0000 7f9b12aba640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c13b4c0 0x7f9b0c1d5290 secure :-1 s=READY pgs=5111 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400caa0 tx=0x7f9b0400cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.150+0000 7f9b02ffd640 10 client.?.objecter ms_handle_connect 0x7f9b0c13b4c0 2023-12-11T20:49:06.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.150+0000 7f9b02ffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:06.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.153+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0400ab40 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.153+0000 7f9b02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:06.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 10 monclient: dump: 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 10 monclient: _finish_auth 0 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0400aca0 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b04012600 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 5 monclient: authenticate success, global_id 21022 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b02ffd640 10 monclient: dump: 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.157 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b10ab6640 4 set_mon_vals no callback set 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 10 monclient: _renew_subs 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c136c70 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 10 client.21022.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 10 monclient: _renew_subs 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c131ae0 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.154+0000 7f9b1451d640 1 librados: init done 2023-12-11T20:49:06.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.155+0000 7f9b1451d640 10 client.21022.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:06.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.155+0000 7f9b1451d640 15 client.21022.objecter _session_command_op_assign -1 1 2023-12-11T20:49:06.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.155+0000 7f9b1451d640 20 client.21022.objecter _get_session osd=-1 returning homeless 2023-12-11T20:49:06.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.155+0000 7f9b1451d640 10 client.21022.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:06.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.155+0000 7f9b1451d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.155+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b0c0af760 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.174+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9b04007070 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.175+0000 7f9b02ffd640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9af006c820 0x7f9af006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.175+0000 7f9b122b9640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9af006c820 0x7f9af006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.175+0000 7f9b122b9640 10 monclient: get_auth_request con 0x7f9af006c820 auth_method 0 2023-12-11T20:49:06.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.175+0000 7f9b02ffd640 10 client.21022.objecter ms_handle_connect 0x7f9af006c820 2023-12-11T20:49:06.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.175+0000 7f9b122b9640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9af006c820 0x7f9af006ece0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9b080017f0 tx=0x7f9b08009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f9b040589c0 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 10 client.21022.objecter ms_dispatch 0x7f9b0c000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:06.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 3 client.21022.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:06.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 3 client.21022.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:06.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 10 client.21022.objecter checking command 1 2023-12-11T20:49:06.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f9af0074560 0x7f9af0076a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 20 client.21022.objecter _get_session s=0x7f9af0073e90 osd=0 3 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 20 client.21022.objecter put_session s=0x7f9af0073e90 osd=0 3 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 15 client.21022.objecter _session_command_op_remove -1 1 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 20 client.21022.objecter _get_session s=0x7f9af0073e90 osd=0 3 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 20 client.21022.objecter get_session s=0x7f9af0073e90 osd=0 3 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 15 client.21022.objecter _session_command_op_assign 0 1 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 20 client.21022.objecter put_session s=0x7f9af0073e90 osd=0 4 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 10 client.21022.objecter _send_command 1 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 --> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9af00770d0 con 0x7f9af0074560 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 20 client.21022.objecter dump_active .. 0 homeless 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b04054d00 con 0x7f9b0c13b4c0 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.178+0000 7f9b02ffd640 10 monclient: handle_get_version_reply finishing 1 version 1057 2023-12-11T20:49:06.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.179+0000 7f9b11ab8640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f9af0074560 0x7f9af0076a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.179+0000 7f9b11ab8640 10 monclient: get_auth_request con 0x7f9af0074560 auth_method 0 2023-12-11T20:49:06.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.182+0000 7f9b02ffd640 10 client.21022.objecter ms_handle_connect 0x7f9af0074560 2023-12-11T20:49:06.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.182+0000 7f9b11ab8640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f9af0074560 0x7f9af0076a00 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.205+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 <== osd.0 v2:172.21.15.86:6820/1914311689 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9af00770d0 con 0x7f9af0074560 2023-12-11T20:49:06.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.205+0000 7f9b02ffd640 10 client.21022.objecter ms_dispatch 0x7f9b0c000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:49:06.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.205+0000 7f9b02ffd640 10 client.21022.objecter _finish_command 1 = system:0 2023-12-11T20:49:06.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.205+0000 7f9b02ffd640 20 client.21022.objecter put_session s=0x7f9af0073e90 osd=0 4 2023-12-11T20:49:06.205 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.205+0000 7f9b02ffd640 15 client.21022.objecter _session_command_op_remove 0 1 2023-12-11T20:49:06.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.220+0000 7f9b1451d640 10 client.21022.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-11T20:49:06.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 15 client.21022.objecter _session_command_op_assign -1 2 2023-12-11T20:49:06.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 20 client.21022.objecter _get_session s=0x7f9af0073e90 osd=0 3 2023-12-11T20:49:06.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 20 client.21022.objecter put_session s=0x7f9af0073e90 osd=0 3 2023-12-11T20:49:06.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 20 client.21022.objecter _get_session s=0x7f9af0073e90 osd=0 3 2023-12-11T20:49:06.220 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 15 client.21022.objecter _session_command_op_remove -1 2 2023-12-11T20:49:06.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 20 client.21022.objecter get_session s=0x7f9af0073e90 osd=0 3 2023-12-11T20:49:06.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 15 client.21022.objecter _session_command_op_assign 0 2 2023-12-11T20:49:06.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 20 client.21022.objecter put_session s=0x7f9af0073e90 osd=0 4 2023-12-11T20:49:06.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 10 client.21022.objecter _send_command 2 2023-12-11T20:49:06.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.221+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 --> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9b0c132dc0 con 0x7f9af0074560 2023-12-11T20:49:06.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b02ffd640 1 -- 172.21.15.86:0/57974444 <== osd.0 v2:172.21.15.86:6820/1914311689 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9b0c132dc0 con 0x7f9af0074560 2023-12-11T20:49:06.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b02ffd640 10 client.21022.objecter ms_dispatch 0x7f9b0c000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:49:06.235 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b02ffd640 10 client.21022.objecter _finish_command 2 = system:0 2023-12-11T20:49:06.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b02ffd640 20 client.21022.objecter put_session s=0x7f9af0073e90 osd=0 4 2023-12-11T20:49:06.236 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b02ffd640 15 client.21022.objecter _session_command_op_remove 0 2 2023-12-11T20:49:06.236 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-12-11T20:49:06.236 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-11T20:49:06.236 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-12-11T20:49:06.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 10 librados: watch_flush enter 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 10 librados: watch_flush exit 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 10 client.21022.objecter close_session for osd.0 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f9af0074560 msgr2=0x7f9af0076a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.86:6820/1914311689,v1:172.21.15.86:6821/1914311689] conn(0x7f9af0074560 0x7f9af0076a00 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 20 client.21022.objecter put_session s=0x7f9af0073e90 osd=0 1 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 20 client.21022.objecter shutdown clearing up homeless session... 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 10 client.21022.objecter successfully canceled tick 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9af006c820 msgr2=0x7f9af006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f9af006c820 0x7f9af006ece0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9b080017f0 tx=0x7f9b08009040 comp rx=0 tx=0).stop 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 10 monclient: shutdown 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:06.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c13b4c0 msgr2=0x7f9b0c1d5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 --2- 172.21.15.86:0/57974444 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b0c13b4c0 0x7f9b0c1d5290 secure :-1 s=READY pgs=5111 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400caa0 tx=0x7f9b0400cea0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b12aba640 1 -- 172.21.15.86:0/57974444 reap_dead start 2023-12-11T20:49:06.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 shutdown_connections 2023-12-11T20:49:06.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 >> 172.21.15.86:0/57974444 conn(0x7f9b0c0b3ef0 msgr2=0x7f9b0c06e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:06.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 shutdown_connections 2023-12-11T20:49:06.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 -- 172.21.15.86:0/57974444 wait complete. 2023-12-11T20:49:06.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.236+0000 7f9b1451d640 1 librados: shutdown 2023-12-11T20:49:06.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-11T20:49:06.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.356+0000 7f855aaa2640 10 monclient: get_monmap_and_config 2023-12-11T20:49:06.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.356+0000 7f855aaa2640 10 monclient: build_initial_monmap 2023-12-11T20:49:06.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.356+0000 7f855aaa2640 10 monclient: monmap: 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:06.358366+0000 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:06.358366+0000 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:06.358 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:06.358 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:06.358 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.357+0000 7f855aaa2640 1 Processor -- start 2023-12-11T20:49:06.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.357+0000 7f855aaa2640 1 -- start start 2023-12-11T20:49:06.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.357+0000 7f855aaa2640 10 monclient: init 2023-12-11T20:49:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-12-11T20:49:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85540a8a20 0x7f85540a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 10 monclient(hunting): picked mon.noname-b con 0x7f85540a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8554145ab0 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f855413f800 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8554145620 con 0x7f8554145ab0 2023-12-11T20:49:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540aace0 con 0x7f855413f800 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85540aafc0 con 0x7f85540a8a20 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855aaa2640 10 monclient(hunting): authenticate will time out at 3432.688381s 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855883e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85540a8a20 0x7f85540a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855903f640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f855413f800 0x7f855414c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:58086/0 (socket says 172.21.15.86:58086) 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.358+0000 7f855903f640 1 -- 172.21.15.86:0/2850365028 learned_addr learned my addr 172.21.15.86:0/2850365028 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.359+0000 7f855883e640 10 monclient(hunting): get_auth_request con 0x7f85540a8a20 auth_method 0 2023-12-11T20:49:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.359+0000 7f855883e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.359+0000 7f855883e640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.359+0000 7f855883e640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.359+0000 7f855883e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.359+0000 7f855883e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.359+0000 7f855883e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 1 -- 172.21.15.86:0/2850365028 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442457844 0 0) 0x7f85540aace0 con 0x7f855413f800 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 10 monclient(hunting): _negotiate 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 10 monclient(hunting): my global_id is 21025 2023-12-11T20:49:06.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 1 -- 172.21.15.86:0/2850365028 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8544003520 con 0x7f855413f800 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 1 -- 172.21.15.86:0/2850365028 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351937073 0 0) 0x7f8554145620 con 0x7f8554145ab0 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 10 monclient(hunting): _negotiate 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 10 monclient(hunting): my global_id is 16878 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f85537fe640 1 -- 172.21.15.86:0/2850365028 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85540aace0 con 0x7f8554145ab0 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f855883e640 10 monclient(hunting): handle_auth_done global_id 19001 payload 274 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f855883e640 1 -- 172.21.15.86:0/2850365028 >> v1:172.21.15.133:6789/0 conn(0x7f855413f800 legacy=0x7f855414c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f855883e640 1 -- 172.21.15.86:0/2850365028 >> v1:172.21.15.86:6789/0 conn(0x7f8554145ab0 legacy=0x7f855413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f855883e640 10 monclient: _finish_hunting 0 2023-12-11T20:49:06.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f855883e640 1 monclient: found mon.noname-b 2023-12-11T20:49:06.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f855883e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:06.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f855883e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:06.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.361+0000 7f855883e640 1 -- 172.21.15.86:0/2850365028 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85540b2210 con 0x7f85540a8a20 2023-12-11T20:49:06.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.363+0000 7f855883e640 1 --2- 172.21.15.86:0/2850365028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85540a8a20 0x7f85540a8e00 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f85480097d0 tx=0x7f854802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c2afc2943ab217 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 1 -- 172.21.15.86:0/2850365028 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548031890 con 0x7f85540a8a20 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: dump: 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.369 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 1 -- 172.21.15.86:0/2850365028 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85480319f0 con 0x7f85540a8a20 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f855aaa2640 5 monclient: authenticate success, global_id 19001 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f855aaa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:06.370 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f855aaa2640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f855aaa2640 4 set_mon_vals no callback set 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 1 -- 172.21.15.86:0/2850365028 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548031d80 con 0x7f85540a8a20 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.366+0000 7f85537fe640 10 monclient: dump: 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 10 monclient: shutdown 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 1 -- 172.21.15.86:0/2850365028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85540a8a20 msgr2=0x7f85540a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 1 --2- 172.21.15.86:0/2850365028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85540a8a20 0x7f85540a8e00 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f85480097d0 tx=0x7f854802b930 comp rx=0 tx=0).stop 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 1 -- 172.21.15.86:0/2850365028 shutdown_connections 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 1 --2- 172.21.15.86:0/2850365028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85540a8a20 0x7f85540a8e00 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 1 -- 172.21.15.86:0/2850365028 >> 172.21.15.86:0/2850365028 conn(0x7f85540b7890 msgr2=0x7f85540b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 1 -- 172.21.15.86:0/2850365028 shutdown_connections 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.367+0000 7f855aaa2640 1 -- 172.21.15.86:0/2850365028 wait complete. 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 10 monclient: build_initial_monmap 2023-12-11T20:49:06.372 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 10 monclient: monmap: 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 1 librados: starting msgr at 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 1 librados: starting objecter 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 1 Processor -- start 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 1 -- start start 2023-12-11T20:49:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 1 librados: setting wanted keys 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 1 librados: calling monclient init 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.368+0000 7f855aaa2640 10 monclient: init 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85540a8a20 0x7f8554140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient(hunting): picked mon.noname-a con 0x7f85540a8a20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f855413f800 0x7f8554143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient(hunting): picked mon.noname-b con 0x7f855413f800 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8554145ab0 0x7f85541416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient(hunting): picked mon.noname-c con 0x7f8554145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85541d7380 con 0x7f85540a8a20 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855413fe90 con 0x7f855413f800 2023-12-11T20:49:06.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85541d7010 con 0x7f8554145ab0 2023-12-11T20:49:06.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:06.375 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855aaa2640 10 monclient(hunting): authenticate will time out at 3432.699637s 2023-12-11T20:49:06.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.369+0000 7f855903f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8554145ab0 0x7f85541416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855883e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85540a8a20 0x7f8554140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f8553fff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f855413f800 0x7f8554143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8554145ab0 0x7f85541416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39332/0 (socket says 172.21.15.86:39332) 2023-12-11T20:49:06.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 1 -- 172.21.15.86:0/2416088855 learned_addr learned my addr 172.21.15.86:0/2416088855 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:06.377 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 10 monclient(hunting): get_auth_request con 0x7f8554145ab0 auth_method 0 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f8553fff640 10 monclient(hunting): get_auth_request con 0x7f855413f800 auth_method 0 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f8553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855883e640 10 monclient(hunting): get_auth_request con 0x7f85540a8a20 auth_method 0 2023-12-11T20:49:06.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855883e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855883e640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855883e640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855903f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855883e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855883e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.370+0000 7f855883e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 10 monclient(hunting): handle_auth_done global_id 19004 payload 824 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 1 -- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f855413f800 msgr2=0x7f8554143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f855413f800 0x7f8554143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 1 -- 172.21.15.86:0/2416088855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85540a8a20 msgr2=0x7f8554140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85540a8a20 0x7f8554140a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 10 monclient: _finish_hunting 0 2023-12-11T20:49:06.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 1 monclient: found mon.noname-c 2023-12-11T20:49:06.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:06.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:49:06.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.373+0000 7f855903f640 1 -- 172.21.15.86:0/2416088855 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8548009400 con 0x7f8554145ab0 2023-12-11T20:49:06.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.375+0000 7f855903f640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8554145ab0 0x7f85541416c0 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f854c00cba0 tx=0x7f854c00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.375+0000 7f85517fa640 10 client.?.objecter ms_handle_connect 0x7f8554145ab0 2023-12-11T20:49:06.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.375+0000 7f85517fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:06.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.378+0000 7f85517fa640 1 -- 172.21.15.86:0/2416088855 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00ace0 con 0x7f8554145ab0 2023-12-11T20:49:06.383 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:06.384 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: dump: 2023-12-11T20:49:06.384 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.384 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.384 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.384 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.384 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: _finish_auth 0 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: _check_auth_tickets 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 1 -- 172.21.15.86:0/2416088855 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c00ae40 con 0x7f8554145ab0 2023-12-11T20:49:06.385 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:06.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 1 -- 172.21.15.86:0/2416088855 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c012620 con 0x7f8554145ab0 2023-12-11T20:49:06.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 5 monclient: authenticate success, global_id 19004 2023-12-11T20:49:06.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:06.386 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f85517fa640 10 monclient: dump: 2023-12-11T20:49:06.386 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.386 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.386 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 10 monclient: _renew_subs 2023-12-11T20:49:06.387 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f8552ffd640 4 set_mon_vals no callback set 2023-12-11T20:49:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8554141c00 con 0x7f8554145ab0 2023-12-11T20:49:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 10 client.19004.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 10 monclient: _renew_subs 2023-12-11T20:49:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85541d79f0 con 0x7f8554145ab0 2023-12-11T20:49:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 1 librados: init done 2023-12-11T20:49:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.379+0000 7f855aaa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:06.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.380+0000 7f855aaa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:06.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.380+0000 7f855aaa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:06.389 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.380+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85541d74f0 con 0x7f8554145ab0 2023-12-11T20:49:06.398 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.398+0000 7f85517fa640 1 -- 172.21.15.86:0/2416088855 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f854c007070 con 0x7f8554145ab0 2023-12-11T20:49:06.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.399+0000 7f85517fa640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f853806c8a0 0x7f853806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.399+0000 7f855883e640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f853806c8a0 0x7f853806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.399+0000 7f855883e640 10 monclient: get_auth_request con 0x7f853806c8a0 auth_method 0 2023-12-11T20:49:06.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.399+0000 7f855883e640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f853806c8a0 0x7f853806ed60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8548036000 tx=0x7f854803c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.400+0000 7f85517fa640 10 client.19004.objecter ms_handle_connect 0x7f853806c8a0 2023-12-11T20:49:06.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.643+0000 7f85517fa640 1 -- 172.21.15.86:0/2416088855 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f854c058800 con 0x7f8554145ab0 2023-12-11T20:49:06.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.643+0000 7f85517fa640 10 client.19004.objecter ms_dispatch 0x7f8554000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:06.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.643+0000 7f85517fa640 3 client.19004.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:06.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.643+0000 7f85517fa640 3 client.19004.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:06.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.643+0000 7f85517fa640 20 client.19004.objecter dump_active .. 0 homeless 2023-12-11T20:49:06.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.643+0000 7f85517fa640 1 -- 172.21.15.86:0/2416088855 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f854c054b40 con 0x7f8554145ab0 2023-12-11T20:49:06.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.643+0000 7f85517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:06.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.643+0000 7f85517fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.812+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f85541d7150 con 0x7f853806c8a0 2023-12-11T20:49:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.813+0000 7f85517fa640 1 -- 172.21.15.86:0/2416088855 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f85541d7150 con 0x7f853806c8a0 2023-12-11T20:49:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 0 to deep-scrub 2023-12-11T20:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 10 librados: watch_flush enter 2023-12-11T20:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 10 librados: watch_flush exit 2023-12-11T20:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 20 client.19004.objecter shutdown clearing up homeless session... 2023-12-11T20:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 10 client.19004.objecter successfully canceled tick 2023-12-11T20:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f853806c8a0 msgr2=0x7f853806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f853806c8a0 0x7f853806ed60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8548036000 tx=0x7f854803c040 comp rx=0 tx=0).stop 2023-12-11T20:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 10 monclient: shutdown 2023-12-11T20:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8554145ab0 msgr2=0x7f85541416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8554145ab0 0x7f85541416c0 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f854c00cba0 tx=0x7f854c00a430 comp rx=0 tx=0).stop 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 shutdown_connections 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f853806c8a0 0x7f853806ed60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8554145ab0 0x7f85541416c0 unknown :-1 s=CLOSED pgs=3333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f855413f800 0x7f8554143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 --2- 172.21.15.86:0/2416088855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85540a8a20 0x7f8554140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 >> 172.21.15.86:0/2416088855 conn(0x7f85540b7890 msgr2=0x7f855413ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 shutdown_connections 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 -- 172.21.15.86:0/2416088855 wait complete. 2023-12-11T20:49:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.815+0000 7f855aaa2640 1 librados: shutdown 2023-12-11T20:49:06.825 INFO:tasks.ceph:Scrubbing osd.1 2023-12-11T20:49:06.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.929+0000 7f587584f640 10 monclient: get_monmap_and_config 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.929+0000 7f587584f640 10 monclient: build_initial_monmap 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.929+0000 7f587584f640 10 monclient: monmap: 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:06.931196+0000 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:06.931196+0000 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:06.929 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.930 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:06.930 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:06.930 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:06.930 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:06.930 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:06.930 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.930+0000 7f587584f640 1 Processor -- start 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.930+0000 7f587584f640 1 -- start start 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.930+0000 7f587584f640 10 monclient: init 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58700a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58700a8e00 0x7f587013f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient(hunting): picked mon.noname-b con 0x7f58700a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f58701423b0 0x7f5870142810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient(hunting): picked mon.noname-c con 0x7f58701423b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58700b21f0 con 0x7f58700a8a20 2023-12-11T20:49:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58700a9cc0 con 0x7f58701423b0 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58700a9ed0 con 0x7f58700a8e00 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f587584f640 10 monclient(hunting): authenticate will time out at 3433.261232s 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f5867fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58700a8e00 0x7f587013f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f5867fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58700a8e00 0x7f587013f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52960/0 (socket says 172.21.15.86:52960) 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f5867fff640 1 -- 172.21.15.86:0/805752471 learned_addr learned my addr 172.21.15.86:0/805752471 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f5867fff640 10 monclient(hunting): get_auth_request con 0x7f58700a8e00 auth_method 0 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f5867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f5867fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.931+0000 7f5867fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.932+0000 7f5867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.932+0000 7f5867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.932+0000 7f5867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.932+0000 7f586f7fe640 1 --2- 172.21.15.86:0/805752471 >> v2:172.21.15.133:3300/0 conn(0x7f58701423b0 0x7f5870142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.932+0000 7f586f7fe640 10 monclient(hunting): get_auth_request con 0x7f58701423b0 auth_method 0 2023-12-11T20:49:06.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.932+0000 7f586f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.932+0000 7f586f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.932+0000 7f586f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.934 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.933+0000 7f586f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.933+0000 7f586f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.933+0000 7f586f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 10 monclient(hunting): handle_auth_done global_id 19007 payload 274 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 1 -- 172.21.15.86:0/805752471 >> v2:172.21.15.133:3300/0 conn(0x7f58701423b0 msgr2=0x7f5870142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 1 --2- 172.21.15.86:0/805752471 >> v2:172.21.15.133:3300/0 conn(0x7f58701423b0 0x7f5870142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 1 -- 172.21.15.86:0/805752471 >> v1:172.21.15.133:6789/0 conn(0x7f58700a8a20 legacy=0x7f587013eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 10 monclient: _finish_hunting 0 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 1 monclient: found mon.noname-b 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.934+0000 7f5867fff640 1 -- 172.21.15.86:0/805752471 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58700aac70 con 0x7f58700a8e00 2023-12-11T20:49:06.936 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.935+0000 7f5867fff640 1 --2- 172.21.15.86:0/805752471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58700a8e00 0x7f587013f5c0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f5858009870 tx=0x7f585802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e960cdf67f646e47 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 1 -- 172.21.15.86:0/805752471 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858031740 con 0x7f58700a8e00 2023-12-11T20:49:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: dump: 2023-12-11T20:49:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: _finish_auth 0 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f587584f640 5 monclient: authenticate success, global_id 19007 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 1 -- 172.21.15.86:0/805752471 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58580318a0 con 0x7f58700a8e00 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f587584f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:06.940 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f587584f640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 1 -- 172.21.15.86:0/805752471 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858031c30 con 0x7f58700a8e00 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f587584f640 4 set_mon_vals no callback set 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.939+0000 7f586e7fc640 10 monclient: dump: 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.941 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 10 monclient: shutdown 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 1 -- 172.21.15.86:0/805752471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58700a8e00 msgr2=0x7f587013f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 1 --2- 172.21.15.86:0/805752471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58700a8e00 0x7f587013f5c0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f5858009870 tx=0x7f585802b7e0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 1 -- 172.21.15.86:0/805752471 shutdown_connections 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 1 --2- 172.21.15.86:0/805752471 >> v2:172.21.15.133:3300/0 conn(0x7f58701423b0 0x7f5870142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 1 --2- 172.21.15.86:0/805752471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58700a8e00 0x7f587013f5c0 unknown :-1 s=CLOSED pgs=3334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 1 -- 172.21.15.86:0/805752471 >> 172.21.15.86:0/805752471 conn(0x7f58700b7870 msgr2=0x7f58700b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 1 -- 172.21.15.86:0/805752471 shutdown_connections 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.940+0000 7f587584f640 1 -- 172.21.15.86:0/805752471 wait complete. 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 10 monclient: build_initial_monmap 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 10 monclient: monmap: 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 1 librados: starting msgr at 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 1 librados: starting objecter 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 1 Processor -- start 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 1 -- start start 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 1 librados: setting wanted keys 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 1 librados: calling monclient init 2023-12-11T20:49:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.941+0000 7f587584f640 10 monclient: init 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58700a8a20 0x7f5870137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient(hunting): picked mon.noname-b con 0x7f58700a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58700a8e00 0x7f5870143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient(hunting): picked mon.noname-a con 0x7f58700a8e00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58701423b0 0x7f58701d75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient(hunting): picked mon.noname-c con 0x7f58701423b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5870144860 con 0x7f58700a8e00 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5870144ad0 con 0x7f58700a8a20 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58701d7090 con 0x7f58701423b0 2023-12-11T20:49:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f587584f640 10 monclient(hunting): authenticate will time out at 3433.272361s 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58701423b0 0x7f58701d75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586effd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58700a8a20 0x7f5870137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f5867fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58700a8e00 0x7f5870143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58701423b0 0x7f58701d75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52972/0 (socket says 172.21.15.86:52972) 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586f7fe640 1 -- 172.21.15.86:0/4039254943 learned_addr learned my addr 172.21.15.86:0/4039254943 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586effd640 10 monclient(hunting): get_auth_request con 0x7f58700a8a20 auth_method 0 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586effd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586f7fe640 10 monclient(hunting): get_auth_request con 0x7f58701423b0 auth_method 0 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.942+0000 7f586f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f5867fff640 10 monclient(hunting): get_auth_request con 0x7f58700a8e00 auth_method 0 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f5867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f5867fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f5867fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f586effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f586effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f586effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f586f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f586f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f586f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f5867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f5867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.943+0000 7f5867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:06.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 10 monclient(hunting): handle_auth_done global_id 21034 payload 824 2023-12-11T20:49:06.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 1 -- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58701423b0 msgr2=0x7f58701d75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58701423b0 0x7f58701d75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 1 -- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58700a8e00 msgr2=0x7f5870143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:06.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58700a8e00 0x7f5870143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 10 monclient: _finish_hunting 0 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 1 monclient: found mon.noname-b 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.945+0000 7f586effd640 1 -- 172.21.15.86:0/4039254943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58580094d0 con 0x7f58700a8a20 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.948+0000 7f586effd640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58700a8a20 0x7f5870137d70 secure :-1 s=READY pgs=5113 cs=0 l=1 rev1=1 crypto rx=0x7f586000c410 tx=0x7f586000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.948+0000 7f586cff9640 10 client.?.objecter ms_handle_connect 0x7f58700a8a20 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.948+0000 7f586cff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58600035f0 con 0x7f58700a8a20 2023-12-11T20:49:06.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f586cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f586cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f586cff9640 10 monclient: dump: 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f586cff9640 10 monclient: _finish_auth 0 2023-12-11T20:49:06.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f586cff9640 10 monclient: _check_auth_tickets 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f586cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 5 monclient: authenticate success, global_id 21034 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 10 monclient: _renew_subs 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5870143b40 con 0x7f58700a8a20 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 10 client.21034.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 10 monclient: _renew_subs 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58701453b0 con 0x7f58700a8a20 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.950+0000 7f587584f640 1 librados: init done 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.951+0000 7f587584f640 10 client.21034.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.951+0000 7f587584f640 15 client.21034.objecter _session_command_op_assign -1 1 2023-12-11T20:49:06.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.951+0000 7f587584f640 20 client.21034.objecter _get_session osd=-1 returning homeless 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.951+0000 7f587584f640 10 client.21034.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.951+0000 7f587584f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.951+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f587013fb50 con 0x7f58700a8a20 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.952+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5860007020 con 0x7f58700a8a20 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.952+0000 7f586cff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.952+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860007410 con 0x7f58700a8a20 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.952+0000 7f586cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.952+0000 7f586cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.952+0000 7f586cff9640 10 monclient: dump: 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:06.955 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:06.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.952+0000 7f586dffb640 4 set_mon_vals no callback set 2023-12-11T20:49:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.972+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5860006790 con 0x7f58700a8a20 2023-12-11T20:49:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.972+0000 7f586cff9640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f585006c820 0x7f585006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.972+0000 7f5867fff640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f585006c820 0x7f585006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.972+0000 7f5867fff640 10 monclient: get_auth_request con 0x7f585006c820 auth_method 0 2023-12-11T20:49:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.973+0000 7f586cff9640 10 client.21034.objecter ms_handle_connect 0x7f585006c820 2023-12-11T20:49:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.973+0000 7f5867fff640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f585006c820 0x7f585006ece0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5858035000 tx=0x7f5858008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f5860058210 con 0x7f58700a8a20 2023-12-11T20:49:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 10 client.21034.objecter ms_dispatch 0x7f5870000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 3 client.21034.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 3 client.21034.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 10 client.21034.objecter checking command 1 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f5850074560 0x7f5850076a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 20 client.21034.objecter _get_session s=0x7f5850073e90 osd=1 3 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 20 client.21034.objecter put_session s=0x7f5850073e90 osd=1 3 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 15 client.21034.objecter _session_command_op_remove -1 1 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 20 client.21034.objecter _get_session s=0x7f5850073e90 osd=1 3 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 20 client.21034.objecter get_session s=0x7f5850073e90 osd=1 3 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 15 client.21034.objecter _session_command_op_assign 1 1 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 20 client.21034.objecter put_session s=0x7f5850073e90 osd=1 4 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 10 client.21034.objecter _send_command 1 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 --> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58500770d0 con 0x7f5850074560 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 20 client.21034.objecter dump_active .. 0 homeless 2023-12-11T20:49:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5860054550 con 0x7f58700a8a20 2023-12-11T20:49:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.974+0000 7f586cff9640 10 monclient: handle_get_version_reply finishing 1 version 1057 2023-12-11T20:49:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.975+0000 7f586f7fe640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f5850074560 0x7f5850076a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.975+0000 7f586f7fe640 10 monclient: get_auth_request con 0x7f5850074560 auth_method 0 2023-12-11T20:49:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.978+0000 7f586f7fe640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f5850074560 0x7f5850076a00 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:06.978+0000 7f586cff9640 10 client.21034.objecter ms_handle_connect 0x7f5850074560 2023-12-11T20:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.001+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 <== osd.1 v2:172.21.15.86:6808/4202858331 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f58500770d0 con 0x7f5850074560 2023-12-11T20:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.001+0000 7f586cff9640 10 client.21034.objecter ms_dispatch 0x7f5870000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.001+0000 7f586cff9640 10 client.21034.objecter _finish_command 1 = system:0 2023-12-11T20:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.001+0000 7f586cff9640 20 client.21034.objecter put_session s=0x7f5850073e90 osd=1 4 2023-12-11T20:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.001+0000 7f586cff9640 15 client.21034.objecter _session_command_op_remove 1 1 2023-12-11T20:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 10 client.21034.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-11T20:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 15 client.21034.objecter _session_command_op_assign -1 2 2023-12-11T20:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 20 client.21034.objecter _get_session s=0x7f5850073e90 osd=1 3 2023-12-11T20:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 20 client.21034.objecter put_session s=0x7f5850073e90 osd=1 3 2023-12-11T20:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 20 client.21034.objecter _get_session s=0x7f5850073e90 osd=1 3 2023-12-11T20:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 15 client.21034.objecter _session_command_op_remove -1 2 2023-12-11T20:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 20 client.21034.objecter get_session s=0x7f5850073e90 osd=1 3 2023-12-11T20:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 15 client.21034.objecter _session_command_op_assign 1 2 2023-12-11T20:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 20 client.21034.objecter put_session s=0x7f5850073e90 osd=1 4 2023-12-11T20:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 10 client.21034.objecter _send_command 2 2023-12-11T20:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.014+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 --> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f587004e840 con 0x7f5850074560 2023-12-11T20:49:07.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.027+0000 7f586cff9640 1 -- 172.21.15.86:0/4039254943 <== osd.1 v2:172.21.15.86:6808/4202858331 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f587004e840 con 0x7f5850074560 2023-12-11T20:49:07.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.027+0000 7f586cff9640 10 client.21034.objecter ms_dispatch 0x7f5870000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:49:07.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.027+0000 7f586cff9640 10 client.21034.objecter _finish_command 2 = system:0 2023-12-11T20:49:07.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.027+0000 7f586cff9640 20 client.21034.objecter put_session s=0x7f5850073e90 osd=1 4 2023-12-11T20:49:07.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.027+0000 7f586cff9640 15 client.21034.objecter _session_command_op_remove 1 2 2023-12-11T20:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-12-11T20:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-11T20:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-12-11T20:49:07.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 10 librados: watch_flush enter 2023-12-11T20:49:07.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 10 librados: watch_flush exit 2023-12-11T20:49:07.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 10 client.21034.objecter close_session for osd.1 2023-12-11T20:49:07.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f5850074560 msgr2=0x7f5850076a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.86:6808/4202858331,v1:172.21.15.86:6809/4202858331] conn(0x7f5850074560 0x7f5850076a00 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 20 client.21034.objecter put_session s=0x7f5850073e90 osd=1 1 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 20 client.21034.objecter shutdown clearing up homeless session... 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 10 client.21034.objecter successfully canceled tick 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f585006c820 msgr2=0x7f585006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f585006c820 0x7f585006ece0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5858035000 tx=0x7f5858008030 comp rx=0 tx=0).stop 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 10 monclient: shutdown 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58700a8a20 msgr2=0x7f5870137d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 --2- 172.21.15.86:0/4039254943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58700a8a20 0x7f5870137d70 secure :-1 s=READY pgs=5113 cs=0 l=1 rev1=1 crypto rx=0x7f586000c410 tx=0x7f586000c810 comp rx=0 tx=0).stop 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f586f7fe640 1 -- 172.21.15.86:0/4039254943 reap_dead start 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 shutdown_connections 2023-12-11T20:49:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 >> 172.21.15.86:0/4039254943 conn(0x7f58700b7870 msgr2=0x7f58700b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 shutdown_connections 2023-12-11T20:49:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 -- 172.21.15.86:0/4039254943 wait complete. 2023-12-11T20:49:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.028+0000 7f587584f640 1 librados: shutdown 2023-12-11T20:49:07.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-11T20:49:07.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.150+0000 7f73597e7640 10 monclient: get_monmap_and_config 2023-12-11T20:49:07.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.150+0000 7f73597e7640 10 monclient: build_initial_monmap 2023-12-11T20:49:07.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.150+0000 7f73597e7640 10 monclient: monmap: 2023-12-11T20:49:07.150 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:07.150 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.150 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:07.152265+0000 2023-12-11T20:49:07.150 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:07.152265+0000 2023-12-11T20:49:07.150 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:07.151 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.151+0000 7f73597e7640 1 Processor -- start 2023-12-11T20:49:07.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.151+0000 7f73597e7640 1 -- start start 2023-12-11T20:49:07.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.151+0000 7f73597e7640 10 monclient: init 2023-12-11T20:49:07.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.151+0000 7f73597e7640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:07.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.151+0000 7f73597e7640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-12-11T20:49:07.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73540a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73540a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 10 monclient(hunting): picked mon.noname-b con 0x7f73541461c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f735413f0f0 0x7f735413f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 10 monclient(hunting): picked mon.noname-a con 0x7f735413f0f0 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7354145620 con 0x7f73540a8a20 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73540aace0 con 0x7f735413f0f0 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540aafc0 con 0x7f73541461c0 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73597e7640 10 monclient(hunting): authenticate will time out at 3433.482217s 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73527fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73540a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f7352ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f73540a8a20 0x7f7354145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:52066/0 (socket says 172.21.15.86:52066) 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f7352ffd640 1 -- 172.21.15.86:0/2176700770 learned_addr learned my addr 172.21.15.86:0/2176700770 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73537fe640 1 --2- 172.21.15.86:0/2176700770 >> v2:172.21.15.86:3300/0 conn(0x7f735413f0f0 0x7f735413f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73527fc640 10 monclient(hunting): get_auth_request con 0x7f73541461c0 auth_method 0 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73527fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.152+0000 7f73527fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73537fe640 10 monclient(hunting): get_auth_request con 0x7f735413f0f0 auth_method 0 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73537fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.153+0000 7f73537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.154+0000 7f7351ffb640 1 -- 172.21.15.86:0/2176700770 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705434156 0 0) 0x7f7354145620 con 0x7f73540a8a20 2023-12-11T20:49:07.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.154+0000 7f7351ffb640 10 monclient(hunting): _negotiate 2023-12-11T20:49:07.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.154+0000 7f7351ffb640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.154+0000 7f7351ffb640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.154+0000 7f7351ffb640 10 monclient(hunting): my global_id is 21037 2023-12-11T20:49:07.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.154+0000 7f7351ffb640 1 -- 172.21.15.86:0/2176700770 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7340003520 con 0x7f73540a8a20 2023-12-11T20:49:07.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.154+0000 7f73527fc640 10 monclient(hunting): handle_auth_done global_id 19013 payload 274 2023-12-11T20:49:07.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.154+0000 7f73527fc640 1 -- 172.21.15.86:0/2176700770 >> v2:172.21.15.86:3300/0 conn(0x7f735413f0f0 msgr2=0x7f735413f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.155+0000 7f73527fc640 1 --2- 172.21.15.86:0/2176700770 >> v2:172.21.15.86:3300/0 conn(0x7f735413f0f0 0x7f735413f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.155+0000 7f73527fc640 1 -- 172.21.15.86:0/2176700770 >> v1:172.21.15.133:6789/0 conn(0x7f73540a8a20 legacy=0x7f7354145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.155+0000 7f73527fc640 10 monclient: _finish_hunting 0 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.155+0000 7f73527fc640 1 monclient: found mon.noname-b 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.155+0000 7f73527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.155+0000 7f73527fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.155+0000 7f73527fc640 1 -- 172.21.15.86:0/2176700770 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735413fac0 con 0x7f73541461c0 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.156+0000 7f73527fc640 1 --2- 172.21.15.86:0/2176700770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73540a8e00 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f733c0097a0 tx=0x7f733c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b7d67c41e7fdf server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f7351ffb640 1 -- 172.21.15.86:0/2176700770 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c031830 con 0x7f73541461c0 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f7351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f7351ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f7351ffb640 10 monclient: dump: 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.159 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f7351ffb640 10 monclient: _finish_auth 0 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f7351ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:49:07.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f7351ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:07.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f73597e7640 5 monclient: authenticate success, global_id 19013 2023-12-11T20:49:07.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.158+0000 7f73597e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:07.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f7351ffb640 1 -- 172.21.15.86:0/2176700770 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c031990 con 0x7f73541461c0 2023-12-11T20:49:07.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f7351ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:07.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f7351ffb640 1 -- 172.21.15.86:0/2176700770 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c031d00 con 0x7f73541461c0 2023-12-11T20:49:07.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 4 set_mon_vals no callback set 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f7351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f7351ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f7351ffb640 10 monclient: dump: 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 10 monclient: shutdown 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 1 -- 172.21.15.86:0/2176700770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 msgr2=0x7f73540a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 1 --2- 172.21.15.86:0/2176700770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73540a8e00 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f733c0097a0 tx=0x7f733c02b8d0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 1 -- 172.21.15.86:0/2176700770 shutdown_connections 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 1 --2- 172.21.15.86:0/2176700770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73540a8e00 unknown :-1 s=CLOSED pgs=3335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 1 --2- 172.21.15.86:0/2176700770 >> v2:172.21.15.86:3300/0 conn(0x7f735413f0f0 0x7f735413f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 1 -- 172.21.15.86:0/2176700770 >> 172.21.15.86:0/2176700770 conn(0x7f73540b7890 msgr2=0x7f73540b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 1 -- 172.21.15.86:0/2176700770 shutdown_connections 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.160+0000 7f73597e7640 1 -- 172.21.15.86:0/2176700770 wait complete. 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.161+0000 7f73597e7640 10 monclient: build_initial_monmap 2023-12-11T20:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.161+0000 7f73597e7640 10 monclient: monmap: 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.161+0000 7f73597e7640 1 librados: starting msgr at 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.161+0000 7f73597e7640 1 librados: starting objecter 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 Processor -- start 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 -- start start 2023-12-11T20:49:07.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 librados: setting wanted keys 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 librados: calling monclient init 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient: init 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73540a8a20 0x7f7354136ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient(hunting): picked mon.noname-b con 0x7f73540a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735413f0f0 0x7f7354137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient(hunting): picked mon.noname-a con 0x7f735413f0f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73541432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient(hunting): picked mon.noname-c con 0x7f73541461c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7354138cf0 con 0x7f735413f0f0 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7354137dc0 con 0x7f73540a8a20 2023-12-11T20:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7354138f40 con 0x7f73541461c0 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.162+0000 7f73597e7640 10 monclient(hunting): authenticate will time out at 3433.492882s 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73537fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73541432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73527fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735413f0f0 0x7f7354137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73537fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73541432c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52990/0 (socket says 172.21.15.86:52990) 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73537fe640 1 -- 172.21.15.86:0/2848354403 learned_addr learned my addr 172.21.15.86:0/2848354403 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73537fe640 10 monclient(hunting): get_auth_request con 0x7f73541461c0 auth_method 0 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73537fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f7352ffd640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73540a8a20 0x7f7354136ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73527fc640 10 monclient(hunting): get_auth_request con 0x7f735413f0f0 auth_method 0 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73527fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f73527fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f7352ffd640 10 monclient(hunting): get_auth_request con 0x7f73540a8a20 auth_method 0 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f7352ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f7352ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.163+0000 7f7352ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f73527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f73527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f73527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f7352ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f7352ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f7352ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f73537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f73537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.164+0000 7f73537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 10 monclient(hunting): handle_auth_done global_id 16890 payload 824 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 1 -- 172.21.15.86:0/2848354403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 msgr2=0x7f73541432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73541432c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 1 -- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73540a8a20 msgr2=0x7f7354136ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73540a8a20 0x7f7354136ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 10 monclient: _finish_hunting 0 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 1 monclient: found mon.noname-a 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f73527fc640 1 -- 172.21.15.86:0/2848354403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c009400 con 0x7f735413f0f0 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.167+0000 7f7352ffd640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73540a8a20 0x7f7354136ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:49:07.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.171+0000 7f73527fc640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735413f0f0 0x7f7354137880 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f73540b4270 tx=0x7f733c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.171+0000 7f734b7fe640 10 client.?.objecter ms_handle_connect 0x7f735413f0f0 2023-12-11T20:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.171+0000 7f734b7fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 1 -- 172.21.15.86:0/2848354403 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c004440 con 0x7f735413f0f0 2023-12-11T20:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: dump: 2023-12-11T20:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 1 -- 172.21.15.86:0/2848354403 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c003610 con 0x7f735413f0f0 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 5 monclient: authenticate success, global_id 16890 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 10 monclient: _renew_subs 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7354139220 con 0x7f735413f0f0 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 1 -- 172.21.15.86:0/2848354403 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c0039c0 con 0x7f735413f0f0 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f734b7fe640 10 monclient: dump: 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.178 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.179 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f7350ff9640 4 set_mon_vals no callback set 2023-12-11T20:49:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 10 client.16890.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 10 monclient: _renew_subs 2023-12-11T20:49:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:49:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7354143800 con 0x7f735413f0f0 2023-12-11T20:49:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.175+0000 7f73597e7640 1 librados: init done 2023-12-11T20:49:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.176+0000 7f73597e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:07.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.176+0000 7f73597e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:07.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.176+0000 7f73597e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:49:07.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.176+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7354137f90 con 0x7f735413f0f0 2023-12-11T20:49:07.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.421+0000 7f734b7fe640 1 -- 172.21.15.86:0/2848354403 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f733c041070 con 0x7f735413f0f0 2023-12-11T20:49:07.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.421+0000 7f734b7fe640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c870 0x7f733406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.421+0000 7f7352ffd640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c870 0x7f733406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.421+0000 7f7352ffd640 10 monclient: get_auth_request con 0x7f733406c870 auth_method 0 2023-12-11T20:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.422+0000 7f7352ffd640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c870 0x7f733406ed30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f734c0011e0 tx=0x7f734c007d30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.422+0000 7f734b7fe640 10 client.16890.objecter ms_handle_connect 0x7f733406c870 2023-12-11T20:49:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.436+0000 7f734b7fe640 1 -- 172.21.15.86:0/2848354403 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f733c07a520 con 0x7f735413f0f0 2023-12-11T20:49:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.436+0000 7f734b7fe640 10 client.16890.objecter ms_dispatch 0x7f7354000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.436+0000 7f734b7fe640 3 client.16890.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.436+0000 7f734b7fe640 3 client.16890.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.436+0000 7f734b7fe640 20 client.16890.objecter dump_active .. 0 homeless 2023-12-11T20:49:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.436+0000 7f734b7fe640 1 -- 172.21.15.86:0/2848354403 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f733c076860 con 0x7f735413f0f0 2023-12-11T20:49:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.436+0000 7f734b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.436+0000 7f734b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.619+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f735414c880 con 0x7f733406c870 2023-12-11T20:49:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.619+0000 7f734b7fe640 1 -- 172.21.15.86:0/2848354403 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f734c0072c0 con 0x7f733406c870 2023-12-11T20:49:07.620 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 1 to deep-scrub 2023-12-11T20:49:07.622 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.621+0000 7f73597e7640 10 librados: watch_flush enter 2023-12-11T20:49:07.622 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.621+0000 7f73597e7640 10 librados: watch_flush exit 2023-12-11T20:49:07.622 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.621+0000 7f73597e7640 20 client.16890.objecter shutdown clearing up homeless session... 2023-12-11T20:49:07.622 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.621+0000 7f73597e7640 10 client.16890.objecter successfully canceled tick 2023-12-11T20:49:07.622 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.621+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c870 msgr2=0x7f733406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.622 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c870 0x7f733406ed30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f734c0011e0 tx=0x7f734c007d30 comp rx=0 tx=0).stop 2023-12-11T20:49:07.622 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 10 monclient: shutdown 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735413f0f0 msgr2=0x7f7354137880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735413f0f0 0x7f7354137880 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f73540b4270 tx=0x7f733c0046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 shutdown_connections 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c870 0x7f733406ed30 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541461c0 0x7f73541432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735413f0f0 0x7f7354137880 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 --2- 172.21.15.86:0/2848354403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73540a8a20 0x7f7354136ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 >> 172.21.15.86:0/2848354403 conn(0x7f73540b7890 msgr2=0x7f735413e640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:07.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 shutdown_connections 2023-12-11T20:49:07.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 -- 172.21.15.86:0/2848354403 wait complete. 2023-12-11T20:49:07.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.622+0000 7f73597e7640 1 librados: shutdown 2023-12-11T20:49:07.633 INFO:tasks.ceph:Scrubbing osd.2 2023-12-11T20:49:07.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-11T20:49:07.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.737+0000 7fecbf1fc640 10 monclient: get_monmap_and_config 2023-12-11T20:49:07.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.737+0000 7fecbf1fc640 10 monclient: build_initial_monmap 2023-12-11T20:49:07.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.737+0000 7fecbf1fc640 10 monclient: monmap: 2023-12-11T20:49:07.737 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:07.737 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.737 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:07.739291+0000 2023-12-11T20:49:07.737 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:07.739291+0000 2023-12-11T20:49:07.737 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:07.738 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.738+0000 7fecbf1fc640 1 Processor -- start 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.738+0000 7fecbf1fc640 1 -- start start 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.738+0000 7fecbf1fc640 10 monclient: init 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7fecb80ae200 0x7fecb806f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 10 monclient(hunting): picked mon.noname-c con 0x7fecb80ae200 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fecb8142160 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecb813c320 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb806fdc0 con 0x7fecb8142160 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb80715b0 con 0x7fecb813c320 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb80ae5e0 con 0x7fecb80ae200 2023-12-11T20:49:07.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecbf1fc640 10 monclient(hunting): authenticate will time out at 3434.069244s 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecb7fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fecb8142160 0x7fecb813bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:60996/0 (socket says 172.21.15.86:60996) 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.739+0000 7fecb7fff640 1 -- 172.21.15.86:0/2926425533 learned_addr learned my addr 172.21.15.86:0/2926425533 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.740+0000 7fecbcf98640 1 --2- 172.21.15.86:0/2926425533 >> v2:172.21.15.133:3300/0 conn(0x7fecb80ae200 0x7fecb806f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.740+0000 7fecbcf98640 10 monclient(hunting): get_auth_request con 0x7fecb80ae200 auth_method 0 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.740+0000 7fecbcf98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.740+0000 7fecbcf98640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.740+0000 7fecbcf98640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.740+0000 7fecbcf98640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.740+0000 7fecbcf98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.740+0000 7fecbcf98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecb77fe640 1 -- 172.21.15.86:0/2926425533 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860514757 0 0) 0x7fecb806fdc0 con 0x7fecb8142160 2023-12-11T20:49:07.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecb77fe640 10 monclient(hunting): _negotiate 2023-12-11T20:49:07.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecb77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecb77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecb77fe640 10 monclient(hunting): my global_id is 16893 2023-12-11T20:49:07.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecb77fe640 1 -- 172.21.15.86:0/2926425533 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca4003520 con 0x7fecb8142160 2023-12-11T20:49:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecbcf98640 10 monclient(hunting): handle_auth_done global_id 21043 payload 274 2023-12-11T20:49:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecbcf98640 1 -- 172.21.15.86:0/2926425533 >> v1:172.21.15.133:6789/0 conn(0x7fecb813c320 legacy=0x7fecb81494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecbcf98640 1 -- 172.21.15.86:0/2926425533 >> v1:172.21.15.86:6789/0 conn(0x7fecb8142160 legacy=0x7fecb813bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecbcf98640 10 monclient: _finish_hunting 0 2023-12-11T20:49:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecbcf98640 1 monclient: found mon.noname-c 2023-12-11T20:49:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecbcf98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecbcf98640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.742+0000 7fecbcf98640 1 -- 172.21.15.86:0/2926425533 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb8071890 con 0x7fecb80ae200 2023-12-11T20:49:07.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.745+0000 7fecbcf98640 1 --2- 172.21.15.86:0/2926425533 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 0x7fecb806f880 secure :-1 s=READY pgs=5115 cs=0 l=1 rev1=1 crypto rx=0x7fecb00097a0 tx=0x7fecb002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70b6db3d270d527e server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 1 -- 172.21.15.86:0/2926425533 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb0031830 con 0x7fecb80ae200 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: dump: 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.750 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 1 -- 172.21.15.86:0/2926425533 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb0031990 con 0x7fecb80ae200 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:07.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 1 -- 172.21.15.86:0/2926425533 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb0031d20 con 0x7fecb80ae200 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecbf1fc640 5 monclient: authenticate success, global_id 21043 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecbf1fc640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecbf1fc640 4 set_mon_vals no callback set 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.748+0000 7fecb77fe640 10 monclient: dump: 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.752 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 10 monclient: shutdown 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 1 -- 172.21.15.86:0/2926425533 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 msgr2=0x7fecb806f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 1 --2- 172.21.15.86:0/2926425533 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 0x7fecb806f880 secure :-1 s=READY pgs=5115 cs=0 l=1 rev1=1 crypto rx=0x7fecb00097a0 tx=0x7fecb002b8d0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 1 -- 172.21.15.86:0/2926425533 shutdown_connections 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 1 --2- 172.21.15.86:0/2926425533 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 0x7fecb806f880 unknown :-1 s=CLOSED pgs=5115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 1 -- 172.21.15.86:0/2926425533 >> 172.21.15.86:0/2926425533 conn(0x7fecb80b3590 msgr2=0x7fecb80b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 1 -- 172.21.15.86:0/2926425533 shutdown_connections 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.749+0000 7fecbf1fc640 1 -- 172.21.15.86:0/2926425533 wait complete. 2023-12-11T20:49:07.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 10 monclient: build_initial_monmap 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 10 monclient: monmap: 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 1 librados: starting msgr at 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 1 librados: starting objecter 2023-12-11T20:49:07.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 1 Processor -- start 2023-12-11T20:49:07.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 1 -- start start 2023-12-11T20:49:07.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 1 librados: setting wanted keys 2023-12-11T20:49:07.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 1 librados: calling monclient init 2023-12-11T20:49:07.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.750+0000 7fecbf1fc640 10 monclient: init 2023-12-11T20:49:07.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:07.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:49:07.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 0x7fecb80ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient(hunting): picked mon.noname-b con 0x7fecb80ae200 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb813c320 0x7fecb80ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient(hunting): picked mon.noname-c con 0x7fecb813c320 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8142160 0x7fecb81d3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient(hunting): picked mon.noname-a con 0x7fecb8142160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb81d2d80 con 0x7fecb8142160 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb813dc00 con 0x7fecb80ae200 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80a9990 con 0x7fecb813c320 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbf1fc640 10 monclient(hunting): authenticate will time out at 3434.081332s 2023-12-11T20:49:07.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbd799640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8142160 0x7fecb81d3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecb7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb813c320 0x7fecb80ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.751+0000 7fecbcf98640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 0x7fecb80ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb813c320 0x7fecb80ac3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53000/0 (socket says 172.21.15.86:53000) 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 1 -- 172.21.15.86:0/496124986 learned_addr learned my addr 172.21.15.86:0/496124986 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 10 monclient(hunting): get_auth_request con 0x7fecb813c320 auth_method 0 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbd799640 10 monclient(hunting): get_auth_request con 0x7fecb8142160 auth_method 0 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbd799640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbd799640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbd799640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbcf98640 10 monclient(hunting): get_auth_request con 0x7fecb80ae200 auth_method 0 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbcf98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.759 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbcf98640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbcf98640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbcf98640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbcf98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbcf98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbd799640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbd799640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecbd799640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.752+0000 7fecb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 10 monclient(hunting): handle_auth_done global_id 21046 payload 824 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 1 -- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb813c320 msgr2=0x7fecb80ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb813c320 0x7fecb80ac3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 1 -- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8142160 msgr2=0x7fecb81d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8142160 0x7fecb81d3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 10 monclient: _finish_hunting 0 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 1 monclient: found mon.noname-b 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.755+0000 7fecbcf98640 1 -- 172.21.15.86:0/496124986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb0009400 con 0x7fecb80ae200 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.756+0000 7fecbcf98640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 0x7fecb80ad0b0 secure :-1 s=READY pgs=5116 cs=0 l=1 rev1=1 crypto rx=0x7fecb0033000 tx=0x7fecb00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.757+0000 7fecb57fa640 10 client.?.objecter ms_handle_connect 0x7fecb80ae200 2023-12-11T20:49:07.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.757+0000 7fecb57fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:07.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb003b9f0 con 0x7fecb80ae200 2023-12-11T20:49:07.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:07.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: dump: 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: _finish_auth 0 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: _check_auth_tickets 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:07.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb003bb70 con 0x7fecb80ae200 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb00453f0 con 0x7fecb80ae200 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 5 monclient: authenticate success, global_id 21046 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb57fa640 10 monclient: dump: 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.764 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 10 monclient: _renew_subs 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb80ac900 con 0x7fecb80ae200 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecb6ffd640 4 set_mon_vals no callback set 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 10 client.21046.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 10 monclient: _renew_subs 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb813cbc0 con 0x7fecb80ae200 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 1 librados: init done 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.760+0000 7fecbf1fc640 10 client.21046.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.761+0000 7fecbf1fc640 15 client.21046.objecter _session_command_op_assign -1 1 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.761+0000 7fecbf1fc640 20 client.21046.objecter _get_session osd=-1 returning homeless 2023-12-11T20:49:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.761+0000 7fecbf1fc640 10 client.21046.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:07.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.761+0000 7fecbf1fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.761+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fecb80708a0 con 0x7fecb80ae200 2023-12-11T20:49:07.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.779+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fecb004e430 con 0x7fecb80ae200 2023-12-11T20:49:07.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.780+0000 7fecb57fa640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fec9806c8a0 0x7fec9806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.780+0000 7fecb7fff640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fec9806c8a0 0x7fec9806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.780+0000 7fecb7fff640 10 monclient: get_auth_request con 0x7fec9806c8a0 auth_method 0 2023-12-11T20:49:07.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.780+0000 7fecb7fff640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fec9806c8a0 0x7fec9806ed60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fecb813d7e0 tx=0x7feca8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.780 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.780+0000 7fecb57fa640 10 client.21046.objecter ms_handle_connect 0x7fec9806c8a0 2023-12-11T20:49:07.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fecb007a980 con 0x7fecb80ae200 2023-12-11T20:49:07.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 10 client.21046.objecter ms_dispatch 0x7fecb8000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:07.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 3 client.21046.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:07.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 3 client.21046.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:07.783 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 10 client.21046.objecter checking command 1 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fec980745e0 0x7fec98076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 20 client.21046.objecter _get_session s=0x7fec98073f10 osd=2 3 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 20 client.21046.objecter put_session s=0x7fec98073f10 osd=2 3 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 15 client.21046.objecter _session_command_op_remove -1 1 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 20 client.21046.objecter _get_session s=0x7fec98073f10 osd=2 3 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 20 client.21046.objecter get_session s=0x7fec98073f10 osd=2 3 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 15 client.21046.objecter _session_command_op_assign 2 1 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 20 client.21046.objecter put_session s=0x7fec98073f10 osd=2 4 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 10 client.21046.objecter _send_command 1 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 --> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec98077150 con 0x7fec980745e0 2023-12-11T20:49:07.784 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.782+0000 7fecb57fa640 20 client.21046.objecter dump_active .. 0 homeless 2023-12-11T20:49:07.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.783+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecb0076cc0 con 0x7fecb80ae200 2023-12-11T20:49:07.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.783+0000 7fecb57fa640 10 monclient: handle_get_version_reply finishing 1 version 1057 2023-12-11T20:49:07.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.783+0000 7fecbd799640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fec980745e0 0x7fec98076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.783+0000 7fecbd799640 10 monclient: get_auth_request con 0x7fec980745e0 auth_method 0 2023-12-11T20:49:07.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.785+0000 7fecbd799640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fec980745e0 0x7fec98076a80 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.785+0000 7fecb57fa640 10 client.21046.objecter ms_handle_connect 0x7fec980745e0 2023-12-11T20:49:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.808+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 <== osd.2 v2:172.21.15.86:6816/406445159 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fec98077150 con 0x7fec980745e0 2023-12-11T20:49:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.808+0000 7fecb57fa640 10 client.21046.objecter ms_dispatch 0x7fecb8000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:49:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.808+0000 7fecb57fa640 10 client.21046.objecter _finish_command 1 = system:0 2023-12-11T20:49:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.808+0000 7fecb57fa640 20 client.21046.objecter put_session s=0x7fec98073f10 osd=2 4 2023-12-11T20:49:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.808+0000 7fecb57fa640 15 client.21046.objecter _session_command_op_remove 2 1 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 10 client.21046.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 15 client.21046.objecter _session_command_op_assign -1 2 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 20 client.21046.objecter _get_session s=0x7fec98073f10 osd=2 3 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 20 client.21046.objecter put_session s=0x7fec98073f10 osd=2 3 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 20 client.21046.objecter _get_session s=0x7fec98073f10 osd=2 3 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 15 client.21046.objecter _session_command_op_remove -1 2 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 20 client.21046.objecter get_session s=0x7fec98073f10 osd=2 3 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 15 client.21046.objecter _session_command_op_assign 2 2 2023-12-11T20:49:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 20 client.21046.objecter put_session s=0x7fec98073f10 osd=2 4 2023-12-11T20:49:07.822 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 10 client.21046.objecter _send_command 2 2023-12-11T20:49:07.822 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.821+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 --> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fecb804e840 con 0x7fec980745e0 2023-12-11T20:49:07.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecb57fa640 1 -- 172.21.15.86:0/496124986 <== osd.2 v2:172.21.15.86:6816/406445159 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fecb804e840 con 0x7fec980745e0 2023-12-11T20:49:07.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecb57fa640 10 client.21046.objecter ms_dispatch 0x7fecb8000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:49:07.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecb57fa640 10 client.21046.objecter _finish_command 2 = system:0 2023-12-11T20:49:07.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecb57fa640 20 client.21046.objecter put_session s=0x7fec98073f10 osd=2 4 2023-12-11T20:49:07.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecb57fa640 15 client.21046.objecter _session_command_op_remove 2 2 2023-12-11T20:49:07.835 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-12-11T20:49:07.835 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-11T20:49:07.835 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 10 librados: watch_flush enter 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 10 librados: watch_flush exit 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 10 client.21046.objecter close_session for osd.2 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fec980745e0 msgr2=0x7fec98076a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.86:6816/406445159,v1:172.21.15.86:6817/406445159] conn(0x7fec980745e0 0x7fec98076a80 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 20 client.21046.objecter put_session s=0x7fec98073f10 osd=2 1 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 20 client.21046.objecter shutdown clearing up homeless session... 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 10 client.21046.objecter successfully canceled tick 2023-12-11T20:49:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fec9806c8a0 msgr2=0x7fec9806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fec9806c8a0 0x7fec9806ed60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fecb813d7e0 tx=0x7feca8009040 comp rx=0 tx=0).stop 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 10 monclient: shutdown 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 msgr2=0x7fecb80ad0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 --2- 172.21.15.86:0/496124986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fecb80ae200 0x7fecb80ad0b0 secure :-1 s=READY pgs=5116 cs=0 l=1 rev1=1 crypto rx=0x7fecb0033000 tx=0x7fecb00046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbd799640 1 -- 172.21.15.86:0/496124986 reap_dead start 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 shutdown_connections 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 >> 172.21.15.86:0/496124986 conn(0x7fecb80b3590 msgr2=0x7fecb813b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 shutdown_connections 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 -- 172.21.15.86:0/496124986 wait complete. 2023-12-11T20:49:07.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.835+0000 7fecbf1fc640 1 librados: shutdown 2023-12-11T20:49:07.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-11T20:49:07.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.948+0000 7f428156c640 10 monclient: get_monmap_and_config 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.948+0000 7f428156c640 10 monclient: build_initial_monmap 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.948+0000 7f428156c640 10 monclient: monmap: 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:07.949864+0000 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:07.949864+0000 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:07.948 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:07.949 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:07.949 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 1 Processor -- start 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 1 -- start start 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient: init 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8a20 0x7f427c13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient(hunting): picked mon.noname-b con 0x7f427c0a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f427c0a8e00 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f427c1423b0 0x7f427c142810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient(hunting): picked mon.noname-c con 0x7f427c1423b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0b21f0 con 0x7f427c0a8e00 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427c0a9cc0 con 0x7f427c1423b0 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427c0a9ed0 con 0x7f427c0a8a20 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.949+0000 7f428156c640 10 monclient(hunting): authenticate will time out at 3434.279910s 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8a20 0x7f427c13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8a20 0x7f427c13eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53010/0 (socket says 172.21.15.86:53010) 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427affd640 1 -- 172.21.15.86:0/3236528514 learned_addr learned my addr 172.21.15.86:0/3236528514 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427affd640 10 monclient(hunting): get_auth_request con 0x7f427c0a8a20 auth_method 0 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427affd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427b7fe640 1 --2- 172.21.15.86:0/3236528514 >> v2:172.21.15.133:3300/0 conn(0x7f427c1423b0 0x7f427c142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427b7fe640 10 monclient(hunting): get_auth_request con 0x7f427c1423b0 auth_method 0 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.950+0000 7f427b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.951+0000 7f427affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.951+0000 7f427affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.951+0000 7f427affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.951+0000 7f427b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.951+0000 7f427b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.951+0000 7f427b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 10 monclient(hunting): handle_auth_done global_id 19022 payload 274 2023-12-11T20:49:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 1 -- 172.21.15.86:0/3236528514 >> v2:172.21.15.133:3300/0 conn(0x7f427c1423b0 msgr2=0x7f427c142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 1 --2- 172.21.15.86:0/3236528514 >> v2:172.21.15.133:3300/0 conn(0x7f427c1423b0 0x7f427c142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 1 -- 172.21.15.86:0/3236528514 >> v1:172.21.15.133:6789/0 conn(0x7f427c0a8e00 legacy=0x7f427c13f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 10 monclient: _finish_hunting 0 2023-12-11T20:49:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 1 monclient: found mon.noname-b 2023-12-11T20:49:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.952+0000 7f427affd640 1 -- 172.21.15.86:0/3236528514 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c0aac70 con 0x7f427c0a8a20 2023-12-11T20:49:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.954+0000 7f427affd640 1 --2- 172.21.15.86:0/3236528514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8a20 0x7f427c13eeb0 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7f426c009870 tx=0x7f426c02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48a889333eef9cd server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 1 -- 172.21.15.86:0/3236528514 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c031740 con 0x7f427c0a8a20 2023-12-11T20:49:07.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 10 monclient: dump: 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 10 monclient: _finish_auth 0 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 1 -- 172.21.15.86:0/3236528514 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c0318a0 con 0x7f427c0a8a20 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f428156c640 5 monclient: authenticate success, global_id 19022 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f428156c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f428156c640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.957+0000 7f427a7fc640 1 -- 172.21.15.86:0/3236528514 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c031c30 con 0x7f427c0a8a20 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 4 set_mon_vals no callback set 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f427a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f427a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f427a7fc640 10 monclient: dump: 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 10 monclient: shutdown 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 1 -- 172.21.15.86:0/3236528514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8a20 msgr2=0x7f427c13eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 1 --2- 172.21.15.86:0/3236528514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8a20 0x7f427c13eeb0 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7f426c009870 tx=0x7f426c02b7e0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 1 -- 172.21.15.86:0/3236528514 shutdown_connections 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 1 --2- 172.21.15.86:0/3236528514 >> v2:172.21.15.133:3300/0 conn(0x7f427c1423b0 0x7f427c142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 1 --2- 172.21.15.86:0/3236528514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8a20 0x7f427c13eeb0 unknown :-1 s=CLOSED pgs=3336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 1 -- 172.21.15.86:0/3236528514 >> 172.21.15.86:0/3236528514 conn(0x7f427c0b7870 msgr2=0x7f427c0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 1 -- 172.21.15.86:0/3236528514 shutdown_connections 2023-12-11T20:49:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.958+0000 7f428156c640 1 -- 172.21.15.86:0/3236528514 wait complete. 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 10 monclient: build_initial_monmap 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 10 monclient: monmap: 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:07.962 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 1 librados: starting msgr at 2023-12-11T20:49:07.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 1 librados: starting objecter 2023-12-11T20:49:07.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 1 Processor -- start 2023-12-11T20:49:07.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 1 -- start start 2023-12-11T20:49:07.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 1 librados: setting wanted keys 2023-12-11T20:49:07.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 1 librados: calling monclient init 2023-12-11T20:49:07.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.959+0000 7f428156c640 10 monclient: init 2023-12-11T20:49:07.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f427c0a8a20 0x7f427c139130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient(hunting): picked mon.noname-b con 0x7f427c0a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8e00 0x7f427c145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient(hunting): picked mon.noname-c con 0x7f427c0a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f427c1423b0 0x7f427c1d74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient(hunting): picked mon.noname-a con 0x7f427c1423b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c1d6ff0 con 0x7f427c1423b0 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c13ac30 con 0x7f427c0a8a20 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427c1460a0 con 0x7f427c0a8e00 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:07.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f428156c640 10 monclient(hunting): authenticate will time out at 3434.290582s 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f4273fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8e00 0x7f427c145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.960+0000 7f427affd640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f427c0a8a20 0x7f427c139130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f427c1423b0 0x7f427c1d74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8e00 0x7f427c145b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53024/0 (socket says 172.21.15.86:53024) 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 1 -- 172.21.15.86:0/698243265 learned_addr learned my addr 172.21.15.86:0/698243265 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 10 monclient(hunting): get_auth_request con 0x7f427c0a8e00 auth_method 0 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427affd640 10 monclient(hunting): get_auth_request con 0x7f427c0a8a20 auth_method 0 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427affd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427b7fe640 10 monclient(hunting): get_auth_request con 0x7f427c1423b0 auth_method 0 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:07.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.961+0000 7f427b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 10 monclient(hunting): handle_auth_done global_id 21052 payload 824 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 1 -- 172.21.15.86:0/698243265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8e00 msgr2=0x7f427c145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8e00 0x7f427c145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 1 -- 172.21.15.86:0/698243265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f427c1423b0 msgr2=0x7f427c1d74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f427c1423b0 0x7f427c1d74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 10 monclient: _finish_hunting 0 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 1 monclient: found mon.noname-b 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f427affd640 1 -- 172.21.15.86:0/698243265 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c0094d0 con 0x7f427c0a8a20 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.964+0000 7f4273fff640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8e00 0x7f427c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.966+0000 7f427affd640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f427c0a8a20 0x7f427c139130 secure :-1 s=READY pgs=5117 cs=0 l=1 rev1=1 crypto rx=0x7f426c033000 tx=0x7f426c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.966+0000 7f42737fe640 10 client.?.objecter ms_handle_connect 0x7f427c0a8a20 2023-12-11T20:49:07.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.966+0000 7f42737fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:07.972 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 1 -- 172.21.15.86:0/698243265 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c033a80 con 0x7f427c0a8a20 2023-12-11T20:49:07.972 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.972 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:07.972 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: dump: 2023-12-11T20:49:07.972 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.972 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.972 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 1 -- 172.21.15.86:0/698243265 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c033c00 con 0x7f427c0a8a20 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 5 monclient: authenticate success, global_id 21052 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 1 -- 172.21.15.86:0/698243265 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c03c400 con 0x7f427c0a8a20 2023-12-11T20:49:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 10 monclient: _renew_subs 2023-12-11T20:49:07.974 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.974 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427c13b050 con 0x7f427c0a8a20 2023-12-11T20:49:07.974 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:07.974 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f4279ffb640 4 set_mon_vals no callback set 2023-12-11T20:49:07.974 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:07.974 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f42737fe640 10 monclient: dump: 2023-12-11T20:49:07.975 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:07.975 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:07.975 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.975 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:07.975 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:07.975 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:07.975 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:07.975 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 10 client.21052.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 10 monclient: _renew_subs 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427c1d7bf0 con 0x7f427c0a8a20 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.970+0000 7f428156c640 1 librados: init done 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.971+0000 7f428156c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.971+0000 7f428156c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.971+0000 7f428156c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.971+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427c1d7160 con 0x7f427c0a8a20 2023-12-11T20:49:07.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.988+0000 7f42737fe640 1 -- 172.21.15.86:0/698243265 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f426c03a040 con 0x7f427c0a8a20 2023-12-11T20:49:07.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.988+0000 7f42737fe640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f425c06c870 0x7f425c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:07.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.989+0000 7f4273fff640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f425c06c870 0x7f425c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:07.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.989+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f425c06c870 auth_method 0 2023-12-11T20:49:07.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.989+0000 7f4273fff640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f425c06c870 0x7f425c06ed30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f427c1db5b0 tx=0x7f4264009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:07.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.989+0000 7f42737fe640 10 client.21052.objecter ms_handle_connect 0x7f425c06c870 2023-12-11T20:49:07.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.991+0000 7f42737fe640 1 -- 172.21.15.86:0/698243265 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f426c07a260 con 0x7f427c0a8a20 2023-12-11T20:49:07.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.991+0000 7f42737fe640 10 client.21052.objecter ms_dispatch 0x7f427c000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:07.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.991+0000 7f42737fe640 3 client.21052.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:07.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.991+0000 7f42737fe640 3 client.21052.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:07.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:07.991+0000 7f42737fe640 20 client.21052.objecter dump_active .. 0 homeless 2023-12-11T20:49:08.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.224+0000 7f42737fe640 1 -- 172.21.15.86:0/698243265 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f426c0765a0 con 0x7f427c0a8a20 2023-12-11T20:49:08.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.224+0000 7f42737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:08.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.224+0000 7f42737fe640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:08.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.393+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f427c141f20 con 0x7f425c06c870 2023-12-11T20:49:08.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.393+0000 7f42737fe640 1 -- 172.21.15.86:0/698243265 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f427c141f20 con 0x7f425c06c870 2023-12-11T20:49:08.393 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 2 to deep-scrub 2023-12-11T20:49:08.395 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 10 librados: watch_flush enter 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 10 librados: watch_flush exit 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 20 client.21052.objecter shutdown clearing up homeless session... 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 10 client.21052.objecter successfully canceled tick 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f425c06c870 msgr2=0x7f425c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f425c06c870 0x7f425c06ed30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f427c1db5b0 tx=0x7f4264009040 comp rx=0 tx=0).stop 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 10 monclient: shutdown 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f427c0a8a20 msgr2=0x7f427c139130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f427c0a8a20 0x7f427c139130 secure :-1 s=READY pgs=5117 cs=0 l=1 rev1=1 crypto rx=0x7f426c033000 tx=0x7f426c0046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 shutdown_connections 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f425c06c870 0x7f425c06ed30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f427c0a8e00 0x7f427c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.396 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f427c1423b0 0x7f427c1d74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 --2- 172.21.15.86:0/698243265 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f427c0a8a20 0x7f427c139130 unknown :-1 s=CLOSED pgs=5117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 >> 172.21.15.86:0/698243265 conn(0x7f427c0b7870 msgr2=0x7f427c13d450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:08.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 shutdown_connections 2023-12-11T20:49:08.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.395+0000 7f428156c640 1 -- 172.21.15.86:0/698243265 wait complete. 2023-12-11T20:49:08.397 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.396+0000 7f428156c640 1 librados: shutdown 2023-12-11T20:49:08.407 INFO:tasks.ceph:Scrubbing osd.3 2023-12-11T20:49:08.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-11T20:49:08.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.512+0000 7f3c4b17c640 10 monclient: get_monmap_and_config 2023-12-11T20:49:08.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.512+0000 7f3c4b17c640 10 monclient: build_initial_monmap 2023-12-11T20:49:08.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.512+0000 7f3c4b17c640 10 monclient: monmap: 2023-12-11T20:49:08.511 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:08.513635+0000 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:08.513635+0000 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:08.512 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.512+0000 7f3c4b17c640 1 Processor -- start 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.512+0000 7f3c4b17c640 1 -- start start 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.512+0000 7f3c4b17c640 10 monclient: init 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c440a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f3c441461c0 0x7f3c440a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient(hunting): picked mon.noname-c con 0x7f3c441461c0 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:08.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c4413f0f0 0x7f3c4413f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient(hunting): picked mon.noname-a con 0x7f3c4413f0f0 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c44145620 con 0x7f3c440a8a20 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c440aace0 con 0x7f3c4413f0f0 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c440aafc0 con 0x7f3c441461c0 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.513+0000 7f3c4b17c640 10 monclient(hunting): authenticate will time out at 3434.843667s 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c4413f0f0 0x7f3c4413f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c4413f0f0 0x7f3c4413f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35250/0 (socket says 172.21.15.86:35250) 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 1 -- 172.21.15.86:0/2472790630 learned_addr learned my addr 172.21.15.86:0/2472790630 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 10 monclient(hunting): get_auth_request con 0x7f3c4413f0f0 auth_method 0 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.516 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:08.516 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.514+0000 7f3c49719640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:08.516 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.515+0000 7f3c49719640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:08.516 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 10 monclient(hunting): handle_auth_done global_id 16902 payload 274 2023-12-11T20:49:08.516 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 1 -- 172.21.15.86:0/2472790630 >> v2:172.21.15.133:3300/0 conn(0x7f3c441461c0 msgr2=0x7f3c440a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:08.516 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 1 --2- 172.21.15.86:0/2472790630 >> v2:172.21.15.133:3300/0 conn(0x7f3c441461c0 0x7f3c440a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.516 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 1 -- 172.21.15.86:0/2472790630 >> v1:172.21.15.133:6789/0 conn(0x7f3c440a8a20 legacy=0x7f3c44145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:08.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 10 monclient: _finish_hunting 0 2023-12-11T20:49:08.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 1 monclient: found mon.noname-a 2023-12-11T20:49:08.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:08.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:49:08.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.516+0000 7f3c49719640 1 -- 172.21.15.86:0/2472790630 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4413fa90 con 0x7f3c4413f0f0 2023-12-11T20:49:08.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.518+0000 7f3c49719640 1 --2- 172.21.15.86:0/2472790630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4413f0f0 0x7f3c4413f550 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f3c440a9a50 tx=0x7f3c3c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4477a35d832f5e5b server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:08.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 1 -- 172.21.15.86:0/2472790630 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c02ca80 con 0x7f3c4413f0f0 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: dump: 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 1 -- 172.21.15.86:0/2472790630 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c02cbe0 con 0x7f3c4413f0f0 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 5 monclient: authenticate success, global_id 16902 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 1 -- 172.21.15.86:0/2472790630 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0343f0 con 0x7f3c4413f0f0 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 4 set_mon_vals no callback set 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c437fe640 10 monclient: dump: 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.523 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.524 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:08.524 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.524 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:08.524 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:08.524 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:08.524 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 10 monclient: shutdown 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 1 -- 172.21.15.86:0/2472790630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4413f0f0 msgr2=0x7f3c4413f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 1 --2- 172.21.15.86:0/2472790630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4413f0f0 0x7f3c4413f550 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f3c440a9a50 tx=0x7f3c3c02bee0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 1 -- 172.21.15.86:0/2472790630 shutdown_connections 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 1 --2- 172.21.15.86:0/2472790630 >> v2:172.21.15.133:3300/0 conn(0x7f3c441461c0 0x7f3c440a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 1 --2- 172.21.15.86:0/2472790630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4413f0f0 0x7f3c4413f550 unknown :-1 s=CLOSED pgs=3355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 1 -- 172.21.15.86:0/2472790630 >> 172.21.15.86:0/2472790630 conn(0x7f3c440b7890 msgr2=0x7f3c440b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 1 -- 172.21.15.86:0/2472790630 shutdown_connections 2023-12-11T20:49:08.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.522+0000 7f3c4b17c640 1 -- 172.21.15.86:0/2472790630 wait complete. 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.523+0000 7f3c4b17c640 10 monclient: build_initial_monmap 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.523+0000 7f3c4b17c640 10 monclient: monmap: 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.523+0000 7f3c4b17c640 1 librados: starting msgr at 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.523+0000 7f3c4b17c640 1 librados: starting objecter 2023-12-11T20:49:08.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 Processor -- start 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 -- start start 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 librados: setting wanted keys 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 librados: calling monclient init 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient: init 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3c440a8a20 0x7f3c44144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient(hunting): picked mon.noname-b con 0x7f3c440a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c4413f0f0 0x7f3c441432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient(hunting): picked mon.noname-c con 0x7f3c4413f0f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c441461c0 0x7f3c441351d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient(hunting): picked mon.noname-a con 0x7f3c441461c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44134d40 con 0x7f3c441461c0 2023-12-11T20:49:08.529 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c441444e0 con 0x7f3c440a8a20 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44144750 con 0x7f3c4413f0f0 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.524+0000 7f3c4b17c640 10 monclient(hunting): authenticate will time out at 3434.855079s 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.525+0000 7f3c43fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c4413f0f0 0x7f3c441432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.525+0000 7f3c43fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c4413f0f0 0x7f3c441432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53034/0 (socket says 172.21.15.86:53034) 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.525+0000 7f3c43fff640 1 -- 172.21.15.86:0/397405111 learned_addr learned my addr 172.21.15.86:0/397405111 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.525+0000 7f3c43fff640 10 monclient(hunting): get_auth_request con 0x7f3c4413f0f0 auth_method 0 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.525+0000 7f3c43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.525+0000 7f3c43fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.525+0000 7f3c43fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.526+0000 7f3c43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:08.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.526+0000 7f3c43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:08.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.526+0000 7f3c43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:08.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.527+0000 7f3c48f18640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3c440a8a20 0x7f3c44144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.527+0000 7f3c48f18640 10 monclient(hunting): get_auth_request con 0x7f3c440a8a20 auth_method 0 2023-12-11T20:49:08.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.527+0000 7f3c48f18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:08.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.527+0000 7f3c48f18640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.527+0000 7f3c48f18640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c48f18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c48f18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c48f18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 10 monclient(hunting): handle_auth_done global_id 19028 payload 824 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 1 -- 172.21.15.86:0/397405111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3c440a8a20 msgr2=0x7f3c44144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3c440a8a20 0x7f3c44144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 1 -- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c441461c0 msgr2=0x7f3c441351d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c441461c0 0x7f3c441351d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 10 monclient: _finish_hunting 0 2023-12-11T20:49:08.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 1 monclient: found mon.noname-c 2023-12-11T20:49:08.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:08.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:49:08.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.528+0000 7f3c43fff640 1 -- 172.21.15.86:0/397405111 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c009b30 con 0x7f3c4413f0f0 2023-12-11T20:49:08.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.531+0000 7f3c43fff640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c4413f0f0 0x7f3c441432f0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7f3c3800b410 tx=0x7f3c3800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:08.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.531+0000 7f3c417fa640 10 client.?.objecter ms_handle_connect 0x7f3c4413f0f0 2023-12-11T20:49:08.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.531+0000 7f3c417fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:08.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38011830 con 0x7f3c4413f0f0 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: dump: 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: _finish_auth 0 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: _check_auth_tickets 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c38011990 con 0x7f3c4413f0f0 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38011d40 con 0x7f3c4413f0f0 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c417fa640 10 monclient: dump: 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.536 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 5 monclient: authenticate success, global_id 19028 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 10 monclient: _renew_subs 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c441349d0 con 0x7f3c4413f0f0 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c42ffd640 4 set_mon_vals no callback set 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 10 client.19028.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 10 monclient: _renew_subs 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c440b5df0 con 0x7f3c4413f0f0 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.534+0000 7f3c4b17c640 1 librados: init done 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.535+0000 7f3c4b17c640 10 client.19028.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.535+0000 7f3c4b17c640 15 client.19028.objecter _session_command_op_assign -1 1 2023-12-11T20:49:08.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.535+0000 7f3c4b17c640 20 client.19028.objecter _get_session osd=-1 returning homeless 2023-12-11T20:49:08.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.535+0000 7f3c4b17c640 10 client.19028.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:08.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.535+0000 7f3c4b17c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:08.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.535+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3c4414c880 con 0x7f3c4413f0f0 2023-12-11T20:49:08.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.555+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3c3801a400 con 0x7f3c4413f0f0 2023-12-11T20:49:08.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.555+0000 7f3c417fa640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3c3006c8a0 0x7f3c3006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.555 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.556+0000 7f3c48f18640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3c3006c8a0 0x7f3c3006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.556 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.556+0000 7f3c48f18640 10 monclient: get_auth_request con 0x7f3c3006c8a0 auth_method 0 2023-12-11T20:49:08.556 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.556+0000 7f3c48f18640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3c3006c8a0 0x7f3c3006ed60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3c340017f0 tx=0x7f3c34009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:08.556 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.556+0000 7f3c417fa640 10 client.19028.objecter ms_handle_connect 0x7f3c3006c8a0 2023-12-11T20:49:08.561 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f3c38059080 con 0x7f3c4413f0f0 2023-12-11T20:49:08.561 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 10 client.19028.objecter ms_dispatch 0x7f3c44000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:08.561 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 3 client.19028.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:08.561 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 3 client.19028.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 10 client.19028.objecter checking command 1 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f3c300745e0 0x7f3c30076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 20 client.19028.objecter _get_session s=0x7f3c30073f10 osd=3 3 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 20 client.19028.objecter put_session s=0x7f3c30073f10 osd=3 3 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 15 client.19028.objecter _session_command_op_remove -1 1 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 20 client.19028.objecter _get_session s=0x7f3c30073f10 osd=3 3 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 20 client.19028.objecter get_session s=0x7f3c30073f10 osd=3 3 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 15 client.19028.objecter _session_command_op_assign 3 1 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 20 client.19028.objecter put_session s=0x7f3c30073f10 osd=3 4 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 10 client.19028.objecter _send_command 1 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 --> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c30077150 con 0x7f3c300745e0 2023-12-11T20:49:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 20 client.19028.objecter dump_active .. 0 homeless 2023-12-11T20:49:08.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c38011ea0 con 0x7f3c4413f0f0 2023-12-11T20:49:08.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.561+0000 7f3c417fa640 10 monclient: handle_get_version_reply finishing 1 version 1057 2023-12-11T20:49:08.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.562+0000 7f3c49719640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f3c300745e0 0x7f3c30076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.562+0000 7f3c49719640 10 monclient: get_auth_request con 0x7f3c300745e0 auth_method 0 2023-12-11T20:49:08.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.565+0000 7f3c49719640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f3c300745e0 0x7f3c30076a80 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:08.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.565+0000 7f3c417fa640 10 client.19028.objecter ms_handle_connect 0x7f3c300745e0 2023-12-11T20:49:08.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.589+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 <== osd.3 v2:172.21.15.86:6800/4064445038 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f3c30077150 con 0x7f3c300745e0 2023-12-11T20:49:08.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.589+0000 7f3c417fa640 10 client.19028.objecter ms_dispatch 0x7f3c44000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:49:08.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.589+0000 7f3c417fa640 10 client.19028.objecter _finish_command 1 = system:0 2023-12-11T20:49:08.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.589+0000 7f3c417fa640 20 client.19028.objecter put_session s=0x7f3c30073f10 osd=3 4 2023-12-11T20:49:08.589 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.590+0000 7f3c417fa640 15 client.19028.objecter _session_command_op_remove 3 1 2023-12-11T20:49:08.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 10 client.19028.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-11T20:49:08.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 15 client.19028.objecter _session_command_op_assign -1 2 2023-12-11T20:49:08.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 20 client.19028.objecter _get_session s=0x7f3c30073f10 osd=3 3 2023-12-11T20:49:08.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 20 client.19028.objecter put_session s=0x7f3c30073f10 osd=3 3 2023-12-11T20:49:08.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 20 client.19028.objecter _get_session s=0x7f3c30073f10 osd=3 3 2023-12-11T20:49:08.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 15 client.19028.objecter _session_command_op_remove -1 2 2023-12-11T20:49:08.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 20 client.19028.objecter get_session s=0x7f3c30073f10 osd=3 3 2023-12-11T20:49:08.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 15 client.19028.objecter _session_command_op_assign 3 2 2023-12-11T20:49:08.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 20 client.19028.objecter put_session s=0x7f3c30073f10 osd=3 4 2023-12-11T20:49:08.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 10 client.19028.objecter _send_command 2 2023-12-11T20:49:08.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.602+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 --> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3c4404e840 con 0x7f3c300745e0 2023-12-11T20:49:08.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c417fa640 1 -- 172.21.15.86:0/397405111 <== osd.3 v2:172.21.15.86:6800/4064445038 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3c4404e840 con 0x7f3c300745e0 2023-12-11T20:49:08.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c417fa640 10 client.19028.objecter ms_dispatch 0x7f3c44000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:49:08.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c417fa640 10 client.19028.objecter _finish_command 2 = system:0 2023-12-11T20:49:08.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c417fa640 20 client.19028.objecter put_session s=0x7f3c30073f10 osd=3 4 2023-12-11T20:49:08.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c417fa640 15 client.19028.objecter _session_command_op_remove 3 2 2023-12-11T20:49:08.616 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-12-11T20:49:08.616 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-11T20:49:08.616 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-12-11T20:49:08.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 10 librados: watch_flush enter 2023-12-11T20:49:08.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 10 librados: watch_flush exit 2023-12-11T20:49:08.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 10 client.19028.objecter close_session for osd.3 2023-12-11T20:49:08.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f3c300745e0 msgr2=0x7f3c30076a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:6800/4064445038,v1:172.21.15.86:6801/4064445038] conn(0x7f3c300745e0 0x7f3c30076a80 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 20 client.19028.objecter put_session s=0x7f3c30073f10 osd=3 1 2023-12-11T20:49:08.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 20 client.19028.objecter shutdown clearing up homeless session... 2023-12-11T20:49:08.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 10 client.19028.objecter successfully canceled tick 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3c3006c8a0 msgr2=0x7f3c3006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f3c3006c8a0 0x7f3c3006ed60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3c340017f0 tx=0x7f3c34009040 comp rx=0 tx=0).stop 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 10 monclient: shutdown 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c4413f0f0 msgr2=0x7f3c441432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c4b17c640 1 --2- 172.21.15.86:0/397405111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c4413f0f0 0x7f3c441432f0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7f3c3800b410 tx=0x7f3c3800b8d0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.616+0000 7f3c49719640 1 -- 172.21.15.86:0/397405111 reap_dead start 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.617+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 shutdown_connections 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.617+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 >> 172.21.15.86:0/397405111 conn(0x7f3c440b7890 msgr2=0x7f3c4413b050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.617+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 shutdown_connections 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.617+0000 7f3c4b17c640 1 -- 172.21.15.86:0/397405111 wait complete. 2023-12-11T20:49:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.617+0000 7f3c4b17c640 1 librados: shutdown 2023-12-11T20:49:08.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-11T20:49:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.727+0000 7f58b38fc640 10 monclient: get_monmap_and_config 2023-12-11T20:49:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.727+0000 7f58b38fc640 10 monclient: build_initial_monmap 2023-12-11T20:49:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.728+0000 7f58b38fc640 10 monclient: monmap: 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:08.729510+0000 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:08.729510+0000 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:08.728 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.728+0000 7f58b38fc640 1 Processor -- start 2023-12-11T20:49:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.728+0000 7f58b38fc640 1 -- start start 2023-12-11T20:49:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.728+0000 7f58b38fc640 10 monclient: init 2023-12-11T20:49:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58ac070b00 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f58ac070250 0x7f58ac140790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient(hunting): picked mon.noname-a con 0x7f58ac070250 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070ee0 0x7f58ac13cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient(hunting): picked mon.noname-b con 0x7f58ac070ee0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac1402b0 con 0x7f58ac070b00 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac0a8a20 con 0x7f58ac070250 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0a8c20 con 0x7f58ac070ee0 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b38fc640 10 monclient(hunting): authenticate will time out at 3435.059620s 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b1698640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f58ac070b00 0x7f58ac0b0230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:52110/0 (socket says 172.21.15.86:52110) 2023-12-11T20:49:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.729+0000 7f58b1698640 1 -- 172.21.15.86:0/3521481788 learned_addr learned my addr 172.21.15.86:0/3521481788 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b1e99640 1 --2- 172.21.15.86:0/3521481788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070ee0 0x7f58ac13cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b0e97640 1 --2- 172.21.15.86:0/3521481788 >> v2:172.21.15.86:3300/0 conn(0x7f58ac070250 0x7f58ac140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b1e99640 10 monclient(hunting): get_auth_request con 0x7f58ac070ee0 auth_method 0 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b1e99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b1e99640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b1e99640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b0e97640 10 monclient(hunting): get_auth_request con 0x7f58ac070250 auth_method 0 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b0e97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b0e97640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b0e97640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b1e99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b1e99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b1e99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b0e97640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b0e97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:08.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.730+0000 7f58b0e97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:08.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58a3fff640 1 -- 172.21.15.86:0/3521481788 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438729529 0 0) 0x7f58ac1402b0 con 0x7f58ac070b00 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58a3fff640 10 monclient(hunting): _negotiate 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58a3fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58a3fff640 10 monclient(hunting): my global_id is 21058 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58a3fff640 1 -- 172.21.15.86:0/3521481788 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5898003520 con 0x7f58ac070b00 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 10 monclient(hunting): handle_auth_done global_id 19031 payload 274 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 1 -- 172.21.15.86:0/3521481788 >> v2:172.21.15.86:3300/0 conn(0x7f58ac070250 msgr2=0x7f58ac140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 1 --2- 172.21.15.86:0/3521481788 >> v2:172.21.15.86:3300/0 conn(0x7f58ac070250 0x7f58ac140790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 1 -- 172.21.15.86:0/3521481788 >> v1:172.21.15.133:6789/0 conn(0x7f58ac070b00 legacy=0x7f58ac0b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 10 monclient: _finish_hunting 0 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 1 monclient: found mon.noname-b 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:08.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:08.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.732+0000 7f58b1e99640 1 -- 172.21.15.86:0/3521481788 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac0a8f00 con 0x7f58ac070ee0 2023-12-11T20:49:08.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.734+0000 7f58b1e99640 1 --2- 172.21.15.86:0/3521481788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070ee0 0x7f58ac13cfa0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7f58a40097a0 tx=0x7f58a402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b201099ffc3c6c07 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:08.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 1 -- 172.21.15.86:0/3521481788 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a402e070 con 0x7f58ac070ee0 2023-12-11T20:49:08.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:08.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:08.737 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: dump: 2023-12-11T20:49:08.737 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:08.737 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.737 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: _finish_auth 0 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: _check_auth_tickets 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 1 -- 172.21.15.86:0/3521481788 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a40315a0 con 0x7f58ac070ee0 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 1 -- 172.21.15.86:0/3521481788 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4031930 con 0x7f58ac070ee0 2023-12-11T20:49:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58b38fc640 5 monclient: authenticate success, global_id 19031 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58b38fc640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58b38fc640 4 set_mon_vals no callback set 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.737+0000 7f58a3fff640 10 monclient: dump: 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:08.739 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 10 monclient: shutdown 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 1 -- 172.21.15.86:0/3521481788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070ee0 msgr2=0x7f58ac13cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 1 --2- 172.21.15.86:0/3521481788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070ee0 0x7f58ac13cfa0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7f58a40097a0 tx=0x7f58a402b840 comp rx=0 tx=0).stop 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 1 -- 172.21.15.86:0/3521481788 shutdown_connections 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 1 --2- 172.21.15.86:0/3521481788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070ee0 0x7f58ac13cfa0 unknown :-1 s=CLOSED pgs=3338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 1 --2- 172.21.15.86:0/3521481788 >> v2:172.21.15.86:3300/0 conn(0x7f58ac070250 0x7f58ac140790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 1 -- 172.21.15.86:0/3521481788 >> 172.21.15.86:0/3521481788 conn(0x7f58ac0b5830 msgr2=0x7f58ac0b5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:08.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 1 -- 172.21.15.86:0/3521481788 shutdown_connections 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.738+0000 7f58b38fc640 1 -- 172.21.15.86:0/3521481788 wait complete. 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 10 monclient: build_initial_monmap 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 10 monclient: monmap: 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 1 librados: starting msgr at 2023-12-11T20:49:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 1 librados: starting objecter 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 1 Processor -- start 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 1 -- start start 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 1 librados: setting wanted keys 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 1 librados: calling monclient init 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.739+0000 7f58b38fc640 10 monclient: init 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58ac070250 0x7f58ac13f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient(hunting): picked mon.noname-a con 0x7f58ac070250 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070b00 0x7f58ac140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient(hunting): picked mon.noname-c con 0x7f58ac070b00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58ac070ee0 0x7f58ac1d81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient(hunting): picked mon.noname-b con 0x7f58ac070ee0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1414b0 con 0x7f58ac070250 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac141720 con 0x7f58ac070ee0 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0a9a60 con 0x7f58ac070b00 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b38fc640 10 monclient(hunting): authenticate will time out at 3435.070515s 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b0e97640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070b00 0x7f58ac140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b1698640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58ac070250 0x7f58ac13f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.740+0000 7f58b1e99640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58ac070ee0 0x7f58ac1d81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070b00 0x7f58ac140f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53054/0 (socket says 172.21.15.86:53054) 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 1 -- 172.21.15.86:0/1581502391 learned_addr learned my addr 172.21.15.86:0/1581502391 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 10 monclient(hunting): get_auth_request con 0x7f58ac070b00 auth_method 0 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1e99640 10 monclient(hunting): get_auth_request con 0x7f58ac070ee0 auth_method 0 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1e99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1e99640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1e99640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1698640 10 monclient(hunting): get_auth_request con 0x7f58ac070250 auth_method 0 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1698640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1698640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1698640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b0e97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1e99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1e99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1e99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1698640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1698640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.741+0000 7f58b1698640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 10 monclient(hunting): handle_auth_done global_id 21061 payload 824 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 1 -- 172.21.15.86:0/1581502391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070b00 msgr2=0x7f58ac140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070b00 0x7f58ac140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 1 -- 172.21.15.86:0/1581502391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58ac070250 msgr2=0x7f58ac13f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58ac070250 0x7f58ac13f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 10 monclient: _finish_hunting 0 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 1 monclient: found mon.noname-b 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b1e99640 1 -- 172.21.15.86:0/1581502391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a4009400 con 0x7f58ac070ee0 2023-12-11T20:49:08.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.744+0000 7f58b0e97640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070b00 0x7f58ac140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:49:08.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.746+0000 7f58b1e99640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58ac070ee0 0x7f58ac1d81a0 secure :-1 s=READY pgs=5119 cs=0 l=1 rev1=1 crypto rx=0x7f58ac140480 tx=0x7f58a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:08.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.746+0000 7f58a1ffb640 10 client.?.objecter ms_handle_connect 0x7f58ac070ee0 2023-12-11T20:49:08.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.746+0000 7f58a1ffb640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 1 -- 172.21.15.86:0/1581502391 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4032840 con 0x7f58ac070ee0 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: dump: 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:08.751 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: _finish_auth 0 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: _check_auth_tickets 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 1 -- 172.21.15.86:0/1581502391 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a40329c0 con 0x7f58ac070ee0 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 1 -- 172.21.15.86:0/1581502391 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4032d70 con 0x7f58ac070ee0 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a1ffb640 10 monclient: dump: 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:08.752 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 5 monclient: authenticate success, global_id 21061 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 10 monclient: _renew_subs 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58a37fe640 4 set_mon_vals no callback set 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac0a9d40 con 0x7f58ac070ee0 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 10 client.21061.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 10 monclient: _renew_subs 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac141e50 con 0x7f58ac070ee0 2023-12-11T20:49:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.750+0000 7f58b38fc640 1 librados: init done 2023-12-11T20:49:08.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.751+0000 7f58b38fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:08.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.751+0000 7f58b38fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:08.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.751+0000 7f58b38fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:08.754 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.751+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac04e840 con 0x7f58ac070ee0 2023-12-11T20:49:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.990+0000 7f58a1ffb640 1 -- 172.21.15.86:0/1581502391 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f58a404e430 con 0x7f58ac070ee0 2023-12-11T20:49:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.990+0000 7f58a1ffb640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f589006c870 0x7f589006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.990+0000 7f58b1698640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f589006c870 0x7f589006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.990+0000 7f58b1698640 10 monclient: get_auth_request con 0x7f589006c870 auth_method 0 2023-12-11T20:49:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.991+0000 7f58b1698640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f589006c870 0x7f589006ed30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f58a8001560 tx=0x7f58a8001a90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.991+0000 7f58a1ffb640 10 client.21061.objecter ms_handle_connect 0x7f589006c870 2023-12-11T20:49:08.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.992+0000 7f58a1ffb640 1 -- 172.21.15.86:0/1581502391 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f58a4031eb0 con 0x7f58ac070ee0 2023-12-11T20:49:08.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.992+0000 7f58a1ffb640 10 client.21061.objecter ms_dispatch 0x7f58ac000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:08.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.992+0000 7f58a1ffb640 3 client.21061.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:08.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.992+0000 7f58a1ffb640 3 client.21061.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:08.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:08.992+0000 7f58a1ffb640 20 client.21061.objecter dump_active .. 0 homeless 2023-12-11T20:49:09.001 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.002+0000 7f58a1ffb640 1 -- 172.21.15.86:0/1581502391 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58a40765b0 con 0x7f58ac070ee0 2023-12-11T20:49:09.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.002+0000 7f58a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:09.002 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.002+0000 7f58a1ffb640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.191+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f58ac141e50 con 0x7f589006c870 2023-12-11T20:49:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.192+0000 7f58a1ffb640 1 -- 172.21.15.86:0/1581502391 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f58ac141e50 con 0x7f589006c870 2023-12-11T20:49:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 3 to deep-scrub 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 10 librados: watch_flush enter 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 10 librados: watch_flush exit 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 20 client.21061.objecter shutdown clearing up homeless session... 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 10 client.21061.objecter successfully canceled tick 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f589006c870 msgr2=0x7f589006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f589006c870 0x7f589006ed30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f58a8001560 tx=0x7f58a8001a90 comp rx=0 tx=0).stop 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 10 monclient: shutdown 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:09.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58ac070ee0 msgr2=0x7f58ac1d81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58ac070ee0 0x7f58ac1d81a0 secure :-1 s=READY pgs=5119 cs=0 l=1 rev1=1 crypto rx=0x7f58ac140480 tx=0x7f58a40046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 shutdown_connections 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f589006c870 0x7f589006ed30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58ac070b00 0x7f58ac140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58ac070ee0 0x7f58ac1d81a0 unknown :-1 s=CLOSED pgs=5119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 --2- 172.21.15.86:0/1581502391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58ac070250 0x7f58ac13f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 >> 172.21.15.86:0/1581502391 conn(0x7f58ac0b5830 msgr2=0x7f58ac0b5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 shutdown_connections 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 -- 172.21.15.86:0/1581502391 wait complete. 2023-12-11T20:49:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.194+0000 7f58b38fc640 1 librados: shutdown 2023-12-11T20:49:09.204 INFO:tasks.ceph:Scrubbing osd.4 2023-12-11T20:49:09.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-11T20:49:09.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.307+0000 7f5cde59e640 10 monclient: get_monmap_and_config 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.307+0000 7f5cde59e640 10 monclient: build_initial_monmap 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.307+0000 7f5cde59e640 10 monclient: monmap: 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:09.308765+0000 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:09.308765+0000 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:09.307 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:09.308 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:09.308 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:09.308 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.307+0000 7f5cde59e640 1 Processor -- start 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 1 -- start start 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient: init 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80b2d10 0x7f5cd80afba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient(hunting): picked mon.noname-b con 0x7f5cd80b2d10 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f5cd80717e0 0x7f5cd806cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient(hunting): picked mon.noname-c con 0x7f5cd80717e0 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5cd813f1d0 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd806c230 con 0x7f5cd813f1d0 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd8071430 con 0x7f5cd80717e0 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd806c480 con 0x7f5cd80b2d10 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.308+0000 7f5cde59e640 10 monclient(hunting): authenticate will time out at 3435.638762s 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80b2d10 0x7f5cd80afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cdcb3b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5cd813f1d0 0x7f5cd80b1240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:32770/0 (socket says 172.21.15.86:32770) 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cdcb3b640 1 -- 172.21.15.86:0/288823622 learned_addr learned my addr 172.21.15.86:0/288823622 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd77fe640 1 --2- 172.21.15.86:0/288823622 >> v2:172.21.15.133:3300/0 conn(0x7f5cd80717e0 0x7f5cd806cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd7fff640 10 monclient(hunting): get_auth_request con 0x7f5cd80b2d10 auth_method 0 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd7fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd77fe640 10 monclient(hunting): get_auth_request con 0x7f5cd80717e0 auth_method 0 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.309+0000 7f5cd77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.310+0000 7f5cd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.310+0000 7f5cd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.310+0000 7f5cd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.310+0000 7f5cd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.310+0000 7f5cd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.310+0000 7f5cd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 10 monclient(hunting): handle_auth_done global_id 21064 payload 274 2023-12-11T20:49:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 1 -- 172.21.15.86:0/288823622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80b2d10 msgr2=0x7f5cd80afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 1 --2- 172.21.15.86:0/288823622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80b2d10 0x7f5cd80afba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 1 -- 172.21.15.86:0/288823622 >> v1:172.21.15.86:6789/0 conn(0x7f5cd813f1d0 legacy=0x7f5cd80b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 10 monclient: _finish_hunting 0 2023-12-11T20:49:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 1 monclient: found mon.noname-c 2023-12-11T20:49:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.311+0000 7f5cd77fe640 1 -- 172.21.15.86:0/288823622 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd8071c40 con 0x7f5cd80717e0 2023-12-11T20:49:09.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.313+0000 7f5cd77fe640 1 --2- 172.21.15.86:0/288823622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd80717e0 0x7f5cd806cf10 secure :-1 s=READY pgs=5120 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0097a0 tx=0x7f5ccc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cb9a9c34f8c8951 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 1 -- 172.21.15.86:0/288823622 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc02e070 con 0x7f5cd80717e0 2023-12-11T20:49:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: dump: 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: _finish_auth 0 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 1 -- 172.21.15.86:0/288823622 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc0315a0 con 0x7f5cd80717e0 2023-12-11T20:49:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 5 monclient: authenticate success, global_id 21064 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 4 set_mon_vals no callback set 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 1 -- 172.21.15.86:0/288823622 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc031930 con 0x7f5cd80717e0 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cd6ffd640 10 monclient: dump: 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.319 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:09.319 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:09.319 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:09.319 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 10 monclient: shutdown 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 1 -- 172.21.15.86:0/288823622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd80717e0 msgr2=0x7f5cd806cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 1 --2- 172.21.15.86:0/288823622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd80717e0 0x7f5cd806cf10 secure :-1 s=READY pgs=5120 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0097a0 tx=0x7f5ccc02b840 comp rx=0 tx=0).stop 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 1 -- 172.21.15.86:0/288823622 shutdown_connections 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 1 --2- 172.21.15.86:0/288823622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd80717e0 0x7f5cd806cf10 unknown :-1 s=CLOSED pgs=5120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 1 --2- 172.21.15.86:0/288823622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80b2d10 0x7f5cd80afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 1 -- 172.21.15.86:0/288823622 >> 172.21.15.86:0/288823622 conn(0x7f5cd80b5e60 msgr2=0x7f5cd80b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 1 -- 172.21.15.86:0/288823622 shutdown_connections 2023-12-11T20:49:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.317+0000 7f5cde59e640 1 -- 172.21.15.86:0/288823622 wait complete. 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.318+0000 7f5cde59e640 10 monclient: build_initial_monmap 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.318+0000 7f5cde59e640 10 monclient: monmap: 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.318+0000 7f5cde59e640 1 librados: starting msgr at 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.318+0000 7f5cde59e640 1 librados: starting objecter 2023-12-11T20:49:09.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.318+0000 7f5cde59e640 1 Processor -- start 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 -- start start 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 librados: setting wanted keys 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 librados: calling monclient init 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient: init 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80717e0 0x7f5cd80aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient(hunting): picked mon.noname-c con 0x7f5cd80717e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cd80b2d10 0x7f5cd8142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient(hunting): picked mon.noname-a con 0x7f5cd80b2d10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd813f1d0 0x7f5cd81da450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient(hunting): picked mon.noname-b con 0x7f5cd813f1d0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80af540 con 0x7f5cd80b2d10 2023-12-11T20:49:09.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80af930 con 0x7f5cd813f1d0 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80af020 con 0x7f5cd80717e0 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.319+0000 7f5cde59e640 10 monclient(hunting): authenticate will time out at 3435.649838s 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80717e0 0x7f5cd80aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cd80b2d10 0x7f5cd8142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cdcb3b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd813f1d0 0x7f5cd81da450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80717e0 0x7f5cd80aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53064/0 (socket says 172.21.15.86:53064) 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 1 -- 172.21.15.86:0/3108774301 learned_addr learned my addr 172.21.15.86:0/3108774301 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 10 monclient(hunting): get_auth_request con 0x7f5cd80717e0 auth_method 0 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cdcb3b640 10 monclient(hunting): get_auth_request con 0x7f5cd813f1d0 auth_method 0 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cdcb3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cdcb3b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cdcb3b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd77fe640 10 monclient(hunting): get_auth_request con 0x7f5cd80b2d10 auth_method 0 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd77fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.320+0000 7f5cd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.321+0000 7f5cdcb3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.321+0000 7f5cdcb3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.321+0000 7f5cdcb3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.321+0000 7f5cd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.326 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.321+0000 7f5cd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.321+0000 7f5cd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 10 monclient(hunting): handle_auth_done global_id 21067 payload 824 2023-12-11T20:49:09.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 1 -- 172.21.15.86:0/3108774301 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80717e0 msgr2=0x7f5cd80aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cd80717e0 0x7f5cd80aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 1 -- 172.21.15.86:0/3108774301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cd80b2d10 msgr2=0x7f5cd8142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cd80b2d10 0x7f5cd8142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 10 monclient: _finish_hunting 0 2023-12-11T20:49:09.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 1 monclient: found mon.noname-b 2023-12-11T20:49:09.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:09.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.323+0000 7f5cdcb3b640 1 -- 172.21.15.86:0/3108774301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc009400 con 0x7f5cd813f1d0 2023-12-11T20:49:09.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.325+0000 7f5cdcb3b640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd813f1d0 0x7f5cd81da450 secure :-1 s=READY pgs=5121 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000cad0 tx=0x7f5cd000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:09.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.325+0000 7f5cd4ff9640 10 client.?.objecter ms_handle_connect 0x7f5cd813f1d0 2023-12-11T20:49:09.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.325+0000 7f5cd4ff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:09.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd000ab70 con 0x7f5cd813f1d0 2023-12-11T20:49:09.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:09.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:09.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: dump: 2023-12-11T20:49:09.329 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: _finish_auth 0 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: _check_auth_tickets 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd000acd0 con 0x7f5cd813f1d0 2023-12-11T20:49:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0012600 con 0x7f5cd813f1d0 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 5 monclient: authenticate success, global_id 21067 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 10 monclient: _renew_subs 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd80ae510 con 0x7f5cd813f1d0 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd4ff9640 10 monclient: dump: 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cd67fc640 4 set_mon_vals no callback set 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 10 client.21067.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 10 monclient: _renew_subs 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd81dab70 con 0x7f5cd813f1d0 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 1 librados: init done 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 10 client.21067.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 15 client.21067.objecter _session_command_op_assign -1 1 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 20 client.21067.objecter _get_session osd=-1 returning homeless 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 10 client.21067.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:09.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.329+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5cd813f640 con 0x7f5cd813f1d0 2023-12-11T20:49:09.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.348+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5cd0007070 con 0x7f5cd813f1d0 2023-12-11T20:49:09.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.348+0000 7f5cd4ff9640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5cc406c820 0x7f5cc406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.349+0000 7f5cd7fff640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5cc406c820 0x7f5cc406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.349+0000 7f5cd7fff640 10 monclient: get_auth_request con 0x7f5cc406c820 auth_method 0 2023-12-11T20:49:09.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.349+0000 7f5cd4ff9640 10 client.21067.objecter ms_handle_connect 0x7f5cc406c820 2023-12-11T20:49:09.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.349+0000 7f5cd7fff640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5cc406c820 0x7f5cc406ece0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009c00 tx=0x7f5cc8009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:09.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f5cd000ae40 con 0x7f5cd813f1d0 2023-12-11T20:49:09.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 10 client.21067.objecter ms_dispatch 0x7f5cd8000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 3 client.21067.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 3 client.21067.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 10 client.21067.objecter checking command 1 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f5cc4074560 0x7f5cc4076a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 20 client.21067.objecter _get_session s=0x7f5cc4073e90 osd=4 3 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 20 client.21067.objecter put_session s=0x7f5cc4073e90 osd=4 3 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 15 client.21067.objecter _session_command_op_remove -1 1 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 20 client.21067.objecter _get_session s=0x7f5cc4073e90 osd=4 3 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 20 client.21067.objecter get_session s=0x7f5cc4073e90 osd=4 3 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 15 client.21067.objecter _session_command_op_assign 4 1 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 20 client.21067.objecter put_session s=0x7f5cc4073e90 osd=4 4 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 10 client.21067.objecter _send_command 1 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 --> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cc40770d0 con 0x7f5cc4074560 2023-12-11T20:49:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.351+0000 7f5cd4ff9640 20 client.21067.objecter dump_active .. 0 homeless 2023-12-11T20:49:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.352+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cd0001ac0 con 0x7f5cd813f1d0 2023-12-11T20:49:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.352+0000 7f5cd4ff9640 10 monclient: handle_get_version_reply finishing 1 version 1057 2023-12-11T20:49:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.352+0000 7f5cd77fe640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f5cc4074560 0x7f5cc4076a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.353+0000 7f5cd77fe640 10 monclient: get_auth_request con 0x7f5cc4074560 auth_method 0 2023-12-11T20:49:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.355+0000 7f5cd4ff9640 10 client.21067.objecter ms_handle_connect 0x7f5cc4074560 2023-12-11T20:49:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.355+0000 7f5cd77fe640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f5cc4074560 0x7f5cc4076a00 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:09.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.379+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 <== osd.4 v2:172.21.15.133:6818/1433273204 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5cc40770d0 con 0x7f5cc4074560 2023-12-11T20:49:09.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.379+0000 7f5cd4ff9640 10 client.21067.objecter ms_dispatch 0x7f5cd8000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:49:09.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.379+0000 7f5cd4ff9640 10 client.21067.objecter _finish_command 1 = system:0 2023-12-11T20:49:09.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.379+0000 7f5cd4ff9640 20 client.21067.objecter put_session s=0x7f5cc4073e90 osd=4 4 2023-12-11T20:49:09.378 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.379+0000 7f5cd4ff9640 15 client.21067.objecter _session_command_op_remove 4 1 2023-12-11T20:49:09.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 10 client.21067.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 15 client.21067.objecter _session_command_op_assign -1 2 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 20 client.21067.objecter _get_session s=0x7f5cc4073e90 osd=4 3 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 20 client.21067.objecter put_session s=0x7f5cc4073e90 osd=4 3 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 20 client.21067.objecter _get_session s=0x7f5cc4073e90 osd=4 3 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 15 client.21067.objecter _session_command_op_remove -1 2 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 20 client.21067.objecter get_session s=0x7f5cc4073e90 osd=4 3 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 15 client.21067.objecter _session_command_op_assign 4 2 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 20 client.21067.objecter put_session s=0x7f5cc4073e90 osd=4 4 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 10 client.21067.objecter _send_command 2 2023-12-11T20:49:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.391+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 --> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5cd80ae860 con 0x7f5cc4074560 2023-12-11T20:49:09.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cd4ff9640 1 -- 172.21.15.86:0/3108774301 <== osd.4 v2:172.21.15.133:6818/1433273204 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5cd80ae860 con 0x7f5cc4074560 2023-12-11T20:49:09.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cd4ff9640 10 client.21067.objecter ms_dispatch 0x7f5cd8000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:49:09.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cd4ff9640 10 client.21067.objecter _finish_command 2 = system:0 2023-12-11T20:49:09.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cd4ff9640 20 client.21067.objecter put_session s=0x7f5cc4073e90 osd=4 4 2023-12-11T20:49:09.404 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cd4ff9640 15 client.21067.objecter _session_command_op_remove 4 2 2023-12-11T20:49:09.404 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-12-11T20:49:09.404 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-11T20:49:09.405 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-12-11T20:49:09.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 10 librados: watch_flush enter 2023-12-11T20:49:09.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 10 librados: watch_flush exit 2023-12-11T20:49:09.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 10 client.21067.objecter close_session for osd.4 2023-12-11T20:49:09.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f5cc4074560 msgr2=0x7f5cc4076a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6818/1433273204,v1:172.21.15.133:6819/1433273204] conn(0x7f5cc4074560 0x7f5cc4076a00 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 20 client.21067.objecter put_session s=0x7f5cc4073e90 osd=4 1 2023-12-11T20:49:09.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 20 client.21067.objecter shutdown clearing up homeless session... 2023-12-11T20:49:09.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 10 client.21067.objecter successfully canceled tick 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5cc406c820 msgr2=0x7f5cc406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5cc406c820 0x7f5cc406ece0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009c00 tx=0x7f5cc8009190 comp rx=0 tx=0).stop 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 10 monclient: shutdown 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd813f1d0 msgr2=0x7f5cd81da450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.404+0000 7f5cde59e640 1 --2- 172.21.15.86:0/3108774301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5cd813f1d0 0x7f5cd81da450 secure :-1 s=READY pgs=5121 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000cad0 tx=0x7f5cd000cf90 comp rx=0 tx=0).stop 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.405+0000 7f5cdcb3b640 1 -- 172.21.15.86:0/3108774301 reap_dead start 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.405+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 shutdown_connections 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.405+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 >> 172.21.15.86:0/3108774301 conn(0x7f5cd80b5e60 msgr2=0x7f5cd813d670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.405+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 shutdown_connections 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.405+0000 7f5cde59e640 1 -- 172.21.15.86:0/3108774301 wait complete. 2023-12-11T20:49:09.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.405+0000 7f5cde59e640 1 librados: shutdown 2023-12-11T20:49:09.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-11T20:49:09.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.517+0000 7f8e1554e640 10 monclient: get_monmap_and_config 2023-12-11T20:49:09.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.517+0000 7f8e1554e640 10 monclient: build_initial_monmap 2023-12-11T20:49:09.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.517+0000 7f8e1554e640 10 monclient: monmap: 2023-12-11T20:49:09.519 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:09.518587+0000 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:09.518587+0000 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.517+0000 7f8e1554e640 1 Processor -- start 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.517+0000 7f8e1554e640 1 -- start start 2023-12-11T20:49:09.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.517+0000 7f8e1554e640 10 monclient: init 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e100a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e100a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e10145ab0 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e1013f800 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e10145620 con 0x7f8e1013f800 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e100aace0 con 0x7f8e10145ab0 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100aafc0 con 0x7f8e100a8a20 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.518+0000 7f8e1554e640 10 monclient(hunting): authenticate will time out at 3435.848492s 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e100a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0e7fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f8e10145ab0 0x7f8e1013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:52122/0 (socket says 172.21.15.86:52122) 2023-12-11T20:49:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0e7fc640 1 -- 172.21.15.86:0/1932299817 learned_addr learned my addr 172.21.15.86:0/1932299817 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:09.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0effd640 10 monclient(hunting): get_auth_request con 0x7f8e100a8a20 auth_method 0 2023-12-11T20:49:09.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0effd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.522 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.519+0000 7f8e0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0effd640 10 monclient(hunting): handle_auth_done global_id 19043 payload 274 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0dffb640 1 -- 172.21.15.86:0/1932299817 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577418499 0 0) 0x7f8e10145620 con 0x7f8e1013f800 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0effd640 1 -- 172.21.15.86:0/1932299817 >> v1:172.21.15.133:6789/0 conn(0x7f8e10145ab0 legacy=0x7f8e1013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0effd640 1 -- 172.21.15.86:0/1932299817 >> v1:172.21.15.86:6789/0 conn(0x7f8e1013f800 legacy=0x7f8e1014c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0effd640 10 monclient: _finish_hunting 0 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0effd640 1 monclient: found mon.noname-b 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0effd640 1 -- 172.21.15.86:0/1932299817 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100b2210 con 0x7f8e100a8a20 2023-12-11T20:49:09.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.521+0000 7f8e0dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-12-11T20:49:09.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.523+0000 7f8e0effd640 1 --2- 172.21.15.86:0/1932299817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e100a8e00 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009dc0 tx=0x7f8e0002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce7e01932b3e11ca server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 1 -- 172.21.15.86:0/1932299817 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0002caa0 con 0x7f8e100a8a20 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: dump: 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: _finish_auth 0 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: _check_auth_tickets 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 1 -- 172.21.15.86:0/1932299817 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0002cc00 con 0x7f8e100a8a20 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e1554e640 5 monclient: authenticate success, global_id 19043 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e1554e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e1554e640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e1554e640 4 set_mon_vals no callback set 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 1 -- 172.21.15.86:0/1932299817 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00034440 con 0x7f8e100a8a20 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.527+0000 7f8e0dffb640 10 monclient: dump: 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:09.529 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 10 monclient: shutdown 2023-12-11T20:49:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 1 -- 172.21.15.86:0/1932299817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 msgr2=0x7f8e100a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 1 --2- 172.21.15.86:0/1932299817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e100a8e00 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009dc0 tx=0x7f8e0002bd70 comp rx=0 tx=0).stop 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 1 -- 172.21.15.86:0/1932299817 shutdown_connections 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 1 --2- 172.21.15.86:0/1932299817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e100a8e00 unknown :-1 s=CLOSED pgs=3339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 1 -- 172.21.15.86:0/1932299817 >> 172.21.15.86:0/1932299817 conn(0x7f8e100b7890 msgr2=0x7f8e100b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 1 -- 172.21.15.86:0/1932299817 shutdown_connections 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 1 -- 172.21.15.86:0/1932299817 wait complete. 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 10 monclient: build_initial_monmap 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 10 monclient: monmap: 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.528+0000 7f8e1554e640 1 librados: starting msgr at 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 librados: starting objecter 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 Processor -- start 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 -- start start 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 librados: setting wanted keys 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 librados: calling monclient init 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient: init 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:49:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e10140ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e100a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e1013f800 0x7f8e101415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1013f800 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e10145ab0 0x7f8e101d7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e10145ab0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e101429a0 con 0x7f8e10145ab0 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e101405b0 con 0x7f8e1013f800 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10141b00 con 0x7f8e100a8a20 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.529+0000 7f8e1554e640 10 monclient(hunting): authenticate will time out at 3435.859967s 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e10140ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e10145ab0 0x7f8e101d7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0e7fc640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e1013f800 0x7f8e101415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e10140ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53082/0 (socket says 172.21.15.86:53082) 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0effd640 1 -- 172.21.15.86:0/2626832568 learned_addr learned my addr 172.21.15.86:0/2626832568 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0effd640 10 monclient(hunting): get_auth_request con 0x7f8e100a8a20 auth_method 0 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0effd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0e7fc640 10 monclient(hunting): get_auth_request con 0x7f8e1013f800 auth_method 0 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f8e10145ab0 auth_method 0 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.530+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:09.537 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.531+0000 7f8e0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.533+0000 7f8e0e7fc640 10 monclient(hunting): handle_auth_done global_id 21073 payload 824 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.533+0000 7f8e0e7fc640 1 -- 172.21.15.86:0/2626832568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 msgr2=0x7f8e10140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.533+0000 7f8e0e7fc640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e10140ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.533+0000 7f8e0e7fc640 1 -- 172.21.15.86:0/2626832568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e10145ab0 msgr2=0x7f8e101d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.533+0000 7f8e0e7fc640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e10145ab0 0x7f8e101d7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.534+0000 7f8e0e7fc640 10 monclient: _finish_hunting 0 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.534+0000 7f8e0e7fc640 1 monclient: found mon.noname-b 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.534+0000 7f8e0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.534+0000 7f8e0e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.534+0000 7f8e0e7fc640 1 -- 172.21.15.86:0/2626832568 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00009a70 con 0x7f8e1013f800 2023-12-11T20:49:09.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.536+0000 7f8e0e7fc640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e1013f800 0x7f8e101415c0 secure :-1 s=READY pgs=5123 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400c4d0 tx=0x7f8e0400c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:09.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.536+0000 7f8df77fe640 10 client.?.objecter ms_handle_connect 0x7f8e1013f800 2023-12-11T20:49:09.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.536+0000 7f8df77fe640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 1 -- 172.21.15.86:0/2626832568 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e040074e0 con 0x7f8e1013f800 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: dump: 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:09.540 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 1 -- 172.21.15.86:0/2626832568 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e04007640 con 0x7f8e1013f800 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 1 -- 172.21.15.86:0/2626832568 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04004a00 con 0x7f8e1013f800 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 5 monclient: authenticate success, global_id 21073 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 10 monclient: _renew_subs 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e10143450 con 0x7f8e1013f800 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8df77fe640 10 monclient: dump: 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e0cff9640 4 set_mon_vals no callback set 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 10 client.21073.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 10 monclient: _renew_subs 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e101d7d40 con 0x7f8e1013f800 2023-12-11T20:49:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.539+0000 7f8e1554e640 1 librados: init done 2023-12-11T20:49:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.540+0000 7f8e1554e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.540+0000 7f8e1554e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.540+0000 7f8e1554e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.540+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1004e840 con 0x7f8e1013f800 2023-12-11T20:49:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.558+0000 7f8df77fe640 1 -- 172.21.15.86:0/2626832568 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8e0401b360 con 0x7f8e1013f800 2023-12-11T20:49:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.558+0000 7f8df77fe640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8dfc06c870 0x7f8dfc06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.558+0000 7f8e0effd640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8dfc06c870 0x7f8dfc06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.558+0000 7f8e0effd640 10 monclient: get_auth_request con 0x7f8dfc06c870 auth_method 0 2023-12-11T20:49:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.559+0000 7f8df77fe640 10 client.21073.objecter ms_handle_connect 0x7f8dfc06c870 2023-12-11T20:49:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.559+0000 7f8e0effd640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8dfc06c870 0x7f8dfc06ed30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8e0002bdb0 tx=0x7f8e00030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.560+0000 7f8df77fe640 1 -- 172.21.15.86:0/2626832568 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f8e04057e80 con 0x7f8e1013f800 2023-12-11T20:49:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.560+0000 7f8df77fe640 10 client.21073.objecter ms_dispatch 0x7f8e10000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.560+0000 7f8df77fe640 3 client.21073.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.560+0000 7f8df77fe640 3 client.21073.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.561+0000 7f8df77fe640 20 client.21073.objecter dump_active .. 0 homeless 2023-12-11T20:49:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.795+0000 7f8df77fe640 1 -- 172.21.15.86:0/2626832568 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e040541c0 con 0x7f8e1013f800 2023-12-11T20:49:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.795+0000 7f8df77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:09.795+0000 7f8df77fe640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:10.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.040+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e101d7fc0 con 0x7f8dfc06c870 2023-12-11T20:49:10.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.041+0000 7f8df77fe640 1 -- 172.21.15.86:0/2626832568 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8e101d7fc0 con 0x7f8dfc06c870 2023-12-11T20:49:10.040 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 4 to deep-scrub 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.042+0000 7f8e1554e640 10 librados: watch_flush enter 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.042+0000 7f8e1554e640 10 librados: watch_flush exit 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.042+0000 7f8e1554e640 20 client.21073.objecter shutdown clearing up homeless session... 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.042+0000 7f8e1554e640 10 client.21073.objecter successfully canceled tick 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.042+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8dfc06c870 msgr2=0x7f8dfc06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8dfc06c870 0x7f8dfc06ed30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8e0002bdb0 tx=0x7f8e00030000 comp rx=0 tx=0).stop 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 10 monclient: shutdown 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e1013f800 msgr2=0x7f8e101415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e1013f800 0x7f8e101415c0 secure :-1 s=READY pgs=5123 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400c4d0 tx=0x7f8e0400c990 comp rx=0 tx=0).stop 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 shutdown_connections 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f8dfc06c870 0x7f8dfc06ed30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e10145ab0 0x7f8e101d7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.043 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e1013f800 0x7f8e101415c0 unknown :-1 s=CLOSED pgs=5123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 --2- 172.21.15.86:0/2626832568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e100a8a20 0x7f8e10140ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 >> 172.21.15.86:0/2626832568 conn(0x7f8e100b7890 msgr2=0x7f8e100b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:10.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 shutdown_connections 2023-12-11T20:49:10.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 -- 172.21.15.86:0/2626832568 wait complete. 2023-12-11T20:49:10.044 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.043+0000 7f8e1554e640 1 librados: shutdown 2023-12-11T20:49:10.053 INFO:tasks.ceph:Scrubbing osd.5 2023-12-11T20:49:10.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.169+0000 7f6a06294640 10 monclient: get_monmap_and_config 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.169+0000 7f6a06294640 10 monclient: build_initial_monmap 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.169+0000 7f6a06294640 10 monclient: monmap: 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:10.170952+0000 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:10.170952+0000 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:10.170 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:10.170 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:10.170 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:10.170 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:10.170 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 1 Processor -- start 2023-12-11T20:49:10.171 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 1 -- start start 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient: init 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a000a8a20 0x7f6a000a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient(hunting): picked mon.noname-b con 0x7f6a000a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a00145ab0 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a0013f800 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a00145620 con 0x7f6a0013f800 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000aace0 con 0x7f6a00145ab0 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a000aafc0 con 0x7f6a000a8a20 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:10.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.170+0000 7f6a06294640 10 monclient(hunting): authenticate will time out at 3436.501054s 2023-12-11T20:49:10.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.171+0000 7f6a04831640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6a0013f800 0x7f6a0014c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:32794/0 (socket says 172.21.15.86:32794) 2023-12-11T20:49:10.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.171+0000 7f6a04831640 1 -- 172.21.15.86:0/1454447565 learned_addr learned my addr 172.21.15.86:0/1454447565 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:10.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.172+0000 7f69fffff640 1 --2- 172.21.15.86:0/1454447565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a000a8a20 0x7f6a000a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.173+0000 7f69fffff640 10 monclient(hunting): get_auth_request con 0x7f6a000a8a20 auth_method 0 2023-12-11T20:49:10.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.173+0000 7f69fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:10.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.173+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.173+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.173+0000 7f69feffd640 1 -- 172.21.15.86:0/1454447565 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979113224 0 0) 0x7f6a00145620 con 0x7f6a0013f800 2023-12-11T20:49:10.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.174+0000 7f69feffd640 10 monclient(hunting): _negotiate 2023-12-11T20:49:10.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.174+0000 7f69feffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.174+0000 7f69feffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.174+0000 7f69feffd640 10 monclient(hunting): my global_id is 16923 2023-12-11T20:49:10.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.174+0000 7f69feffd640 1 -- 172.21.15.86:0/1454447565 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e8003520 con 0x7f6a0013f800 2023-12-11T20:49:10.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.174+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:10.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.174+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:10.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.174+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.175+0000 7f69feffd640 1 -- 172.21.15.86:0/1454447565 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687945567 0 0) 0x7f6a000aace0 con 0x7f6a00145ab0 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.175+0000 7f69feffd640 10 monclient(hunting): _negotiate 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.175+0000 7f69feffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.175+0000 7f69feffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.175+0000 7f69feffd640 10 monclient(hunting): my global_id is 21076 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.175+0000 7f69feffd640 1 -- 172.21.15.86:0/1454447565 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a00145620 con 0x7f6a00145ab0 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.176+0000 7f69fffff640 10 monclient(hunting): handle_auth_done global_id 19049 payload 274 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.176+0000 7f69fffff640 1 -- 172.21.15.86:0/1454447565 >> v1:172.21.15.133:6789/0 conn(0x7f6a00145ab0 legacy=0x7f6a0013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.176+0000 7f69fffff640 1 -- 172.21.15.86:0/1454447565 >> v1:172.21.15.86:6789/0 conn(0x7f6a0013f800 legacy=0x7f6a0014c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.176+0000 7f69fffff640 10 monclient: _finish_hunting 0 2023-12-11T20:49:10.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.176+0000 7f69fffff640 1 monclient: found mon.noname-b 2023-12-11T20:49:10.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.176+0000 7f69fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:10.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.176+0000 7f69fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:10.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.176+0000 7f69fffff640 1 -- 172.21.15.86:0/1454447565 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a000b2210 con 0x7f6a000a8a20 2023-12-11T20:49:10.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.179+0000 7f69fffff640 1 --2- 172.21.15.86:0/1454447565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a000a8a20 0x7f6a000a8e00 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7f69f00097d0 tx=0x7f69f002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5243e4e685ffbc8 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:10.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 1 -- 172.21.15.86:0/1454447565 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f0031890 con 0x7f6a000a8a20 2023-12-11T20:49:10.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:10.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:10.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: dump: 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: _finish_auth 0 2023-12-11T20:49:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 1 -- 172.21.15.86:0/1454447565 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f00319f0 con 0x7f6a000a8a20 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 5 monclient: authenticate success, global_id 19049 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 1 -- 172.21.15.86:0/1454447565 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f0031d80 con 0x7f6a000a8a20 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 4 set_mon_vals no callback set 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f69feffd640 10 monclient: dump: 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.186 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.186 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.186 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:10.186 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.186 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:10.186 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:10.186 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:10.186 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 10 monclient: shutdown 2023-12-11T20:49:10.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:10.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 1 -- 172.21.15.86:0/1454447565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a000a8a20 msgr2=0x7f6a000a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 1 --2- 172.21.15.86:0/1454447565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a000a8a20 0x7f6a000a8e00 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7f69f00097d0 tx=0x7f69f002b930 comp rx=0 tx=0).stop 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 1 -- 172.21.15.86:0/1454447565 shutdown_connections 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 1 --2- 172.21.15.86:0/1454447565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a000a8a20 0x7f6a000a8e00 unknown :-1 s=CLOSED pgs=3340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 1 -- 172.21.15.86:0/1454447565 >> 172.21.15.86:0/1454447565 conn(0x7f6a000b7890 msgr2=0x7f6a000b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 1 -- 172.21.15.86:0/1454447565 shutdown_connections 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.184+0000 7f6a06294640 1 -- 172.21.15.86:0/1454447565 wait complete. 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.185+0000 7f6a06294640 10 monclient: build_initial_monmap 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.185+0000 7f6a06294640 10 monclient: monmap: 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:10.190 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.185+0000 7f6a06294640 1 librados: starting msgr at 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.185+0000 7f6a06294640 1 librados: starting objecter 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.185+0000 7f6a06294640 1 Processor -- start 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 -- start start 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 librados: setting wanted keys 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 librados: calling monclient init 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient: init 2023-12-11T20:49:10.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a000a8a20 0x7f6a00142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient(hunting): picked mon.noname-b con 0x7f6a000a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0013f800 0x7f6a001439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0013f800 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a00145ab0 0x7f6a001d7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient(hunting): picked mon.noname-c con 0x7f6a00145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a001d7100 con 0x7f6a0013f800 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:10.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00143f20 con 0x7f6a000a8a20 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00143260 con 0x7f6a00145ab0 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.186+0000 7f6a06294640 10 monclient(hunting): authenticate will time out at 3436.516790s 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a00145ab0 0x7f6a001d7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0013f800 0x7f6a001439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69fffff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a000a8a20 0x7f6a00142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a00145ab0 0x7f6a001d7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53106/0 (socket says 172.21.15.86:53106) 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 1 -- 172.21.15.86:0/1408637584 learned_addr learned my addr 172.21.15.86:0/1408637584 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 10 monclient(hunting): get_auth_request con 0x7f6a00145ab0 auth_method 0 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69fffff640 10 monclient(hunting): get_auth_request con 0x7f6a000a8a20 auth_method 0 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69ff7fe640 10 monclient(hunting): get_auth_request con 0x7f6a0013f800 auth_method 0 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69ff7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f69ff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.187+0000 7f6a04831640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:10.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.188+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:10.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.188+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:10.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.188+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:10.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.188+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:10.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.188+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:10.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.188+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:10.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 10 monclient(hunting): handle_auth_done global_id 19052 payload 824 2023-12-11T20:49:10.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 1 -- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a000a8a20 msgr2=0x7f6a00142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a000a8a20 0x7f6a00142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 1 -- 172.21.15.86:0/1408637584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0013f800 msgr2=0x7f6a001439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0013f800 0x7f6a001439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 10 monclient: _finish_hunting 0 2023-12-11T20:49:10.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 1 monclient: found mon.noname-c 2023-12-11T20:49:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:49:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f6a04831640 1 -- 172.21.15.86:0/1408637584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0009400 con 0x7f6a00145ab0 2023-12-11T20:49:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.190+0000 7f69fffff640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a000a8a20 0x7f6a00142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-11T20:49:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.193+0000 7f6a04831640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a00145ab0 0x7f6a001d7520 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7f69f800cb70 tx=0x7f69f800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.193+0000 7f69fcff9640 10 client.?.objecter ms_handle_connect 0x7f6a00145ab0 2023-12-11T20:49:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.193+0000 7f69fcff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f800ad40 con 0x7f6a00145ab0 2023-12-11T20:49:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: dump: 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: _finish_auth 0 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: _check_auth_tickets 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f8012400 con 0x7f6a00145ab0 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80127b0 con 0x7f6a00145ab0 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 5 monclient: authenticate success, global_id 19052 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 10 monclient: _renew_subs 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a00144ff0 con 0x7f6a00145ab0 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fe7fc640 4 set_mon_vals no callback set 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f69fcff9640 10 monclient: dump: 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 10 client.19052.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 10 monclient: _renew_subs 2023-12-11T20:49:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a00140350 con 0x7f6a00145ab0 2023-12-11T20:49:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.197+0000 7f6a06294640 1 librados: init done 2023-12-11T20:49:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.198+0000 7f6a06294640 10 client.19052.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.198+0000 7f6a06294640 15 client.19052.objecter _session_command_op_assign -1 1 2023-12-11T20:49:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.198+0000 7f6a06294640 20 client.19052.objecter _get_session osd=-1 returning homeless 2023-12-11T20:49:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.198+0000 7f6a06294640 10 client.19052.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.198+0000 7f6a06294640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.198+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a001d7240 con 0x7f6a00145ab0 2023-12-11T20:49:10.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.223+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f69f8007070 con 0x7f6a00145ab0 2023-12-11T20:49:10.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.223+0000 7f69fcff9640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f69ec06c870 0x7f69ec06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.224+0000 7f69fffff640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f69ec06c870 0x7f69ec06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.223 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.224+0000 7f69fffff640 10 monclient: get_auth_request con 0x7f69ec06c870 auth_method 0 2023-12-11T20:49:10.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.225+0000 7f69fcff9640 10 client.19052.objecter ms_handle_connect 0x7f69ec06c870 2023-12-11T20:49:10.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.225+0000 7f69fffff640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f69ec06c870 0x7f69ec06ed30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f69f0036000 tx=0x7f69f003c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:10.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.227+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f69f8058a80 con 0x7f6a00145ab0 2023-12-11T20:49:10.227 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.227+0000 7f69fcff9640 10 client.19052.objecter ms_dispatch 0x7f6a00000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.227+0000 7f69fcff9640 3 client.19052.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.227+0000 7f69fcff9640 3 client.19052.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 10 client.19052.objecter checking command 1 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f69ec0745b0 0x7f69ec076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 20 client.19052.objecter _get_session s=0x7f69ec073ee0 osd=5 3 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 20 client.19052.objecter put_session s=0x7f69ec073ee0 osd=5 3 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 15 client.19052.objecter _session_command_op_remove -1 1 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 20 client.19052.objecter _get_session s=0x7f69ec073ee0 osd=5 3 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 20 client.19052.objecter get_session s=0x7f69ec073ee0 osd=5 3 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 15 client.19052.objecter _session_command_op_assign 5 1 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 20 client.19052.objecter put_session s=0x7f69ec073ee0 osd=5 4 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 10 client.19052.objecter _send_command 1 2023-12-11T20:49:10.228 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 --> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f69ec077120 con 0x7f69ec0745b0 2023-12-11T20:49:10.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 20 client.19052.objecter dump_active .. 0 homeless 2023-12-11T20:49:10.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69f8001c20 con 0x7f6a00145ab0 2023-12-11T20:49:10.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.228+0000 7f69fcff9640 10 monclient: handle_get_version_reply finishing 1 version 1057 2023-12-11T20:49:10.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.229+0000 7f69ff7fe640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f69ec0745b0 0x7f69ec076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.229+0000 7f69ff7fe640 10 monclient: get_auth_request con 0x7f69ec0745b0 auth_method 0 2023-12-11T20:49:10.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.232+0000 7f69fcff9640 10 client.19052.objecter ms_handle_connect 0x7f69ec0745b0 2023-12-11T20:49:10.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.232+0000 7f69ff7fe640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f69ec0745b0 0x7f69ec076a50 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:10.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.264+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 <== osd.5 v2:172.21.15.133:6802/3582621102 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f69ec077120 con 0x7f69ec0745b0 2023-12-11T20:49:10.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.264+0000 7f69fcff9640 10 client.19052.objecter ms_dispatch 0x7f6a00000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:49:10.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.264+0000 7f69fcff9640 10 client.19052.objecter _finish_command 1 = system:0 2023-12-11T20:49:10.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.264+0000 7f69fcff9640 20 client.19052.objecter put_session s=0x7f69ec073ee0 osd=5 4 2023-12-11T20:49:10.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.264+0000 7f69fcff9640 15 client.19052.objecter _session_command_op_remove 5 1 2023-12-11T20:49:10.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 10 client.19052.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-11T20:49:10.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 15 client.19052.objecter _session_command_op_assign -1 2 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 20 client.19052.objecter _get_session s=0x7f69ec073ee0 osd=5 3 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 20 client.19052.objecter put_session s=0x7f69ec073ee0 osd=5 3 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 20 client.19052.objecter _get_session s=0x7f69ec073ee0 osd=5 3 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 15 client.19052.objecter _session_command_op_remove -1 2 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 20 client.19052.objecter get_session s=0x7f69ec073ee0 osd=5 3 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 15 client.19052.objecter _session_command_op_assign 5 2 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 20 client.19052.objecter put_session s=0x7f69ec073ee0 osd=5 4 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 10 client.19052.objecter _send_command 2 2023-12-11T20:49:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.278+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 --> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6a0004e840 con 0x7f69ec0745b0 2023-12-11T20:49:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f69fcff9640 1 -- 172.21.15.86:0/1408637584 <== osd.5 v2:172.21.15.133:6802/3582621102 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6a0004e840 con 0x7f69ec0745b0 2023-12-11T20:49:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f69fcff9640 10 client.19052.objecter ms_dispatch 0x7f6a00000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:49:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f69fcff9640 10 client.19052.objecter _finish_command 2 = system:0 2023-12-11T20:49:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f69fcff9640 20 client.19052.objecter put_session s=0x7f69ec073ee0 osd=5 4 2023-12-11T20:49:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f69fcff9640 15 client.19052.objecter _session_command_op_remove 5 2 2023-12-11T20:49:10.291 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-12-11T20:49:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-11T20:49:10.291 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-12-11T20:49:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 10 librados: watch_flush enter 2023-12-11T20:49:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 10 librados: watch_flush exit 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 10 client.19052.objecter close_session for osd.5 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f69ec0745b0 msgr2=0x7f69ec076a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6802/3582621102,v1:172.21.15.133:6803/3582621102] conn(0x7f69ec0745b0 0x7f69ec076a50 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 20 client.19052.objecter put_session s=0x7f69ec073ee0 osd=5 1 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 20 client.19052.objecter shutdown clearing up homeless session... 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 10 client.19052.objecter successfully canceled tick 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f69ec06c870 msgr2=0x7f69ec06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f69ec06c870 0x7f69ec06ed30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f69f0036000 tx=0x7f69f003c040 comp rx=0 tx=0).stop 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 10 monclient: shutdown 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a00145ab0 msgr2=0x7f6a001d7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 --2- 172.21.15.86:0/1408637584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a00145ab0 0x7f6a001d7520 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7f69f800cb70 tx=0x7f69f800a430 comp rx=0 tx=0).stop 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a04831640 1 -- 172.21.15.86:0/1408637584 reap_dead start 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 shutdown_connections 2023-12-11T20:49:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 >> 172.21.15.86:0/1408637584 conn(0x7f6a000b7890 msgr2=0x7f6a000b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 shutdown_connections 2023-12-11T20:49:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 -- 172.21.15.86:0/1408637584 wait complete. 2023-12-11T20:49:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.291+0000 7f6a06294640 1 librados: shutdown 2023-12-11T20:49:10.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-11T20:49:10.403 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.404+0000 7fdbc4490640 10 monclient: get_monmap_and_config 2023-12-11T20:49:10.403 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.404+0000 7fdbc4490640 10 monclient: build_initial_monmap 2023-12-11T20:49:10.403 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.404+0000 7fdbc4490640 10 monclient: monmap: 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:10.405624+0000 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:10.405624+0000 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:10.404 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.404+0000 7fdbc4490640 1 Processor -- start 2023-12-11T20:49:10.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.404+0000 7fdbc4490640 1 -- start start 2023-12-11T20:49:10.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.404+0000 7fdbc4490640 10 monclient: init 2023-12-11T20:49:10.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:10.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-12-11T20:49:10.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbbc0a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:10.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdbbc0a8e00 0x7fdbbc13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 10 monclient(hunting): picked mon.noname-a con 0x7fdbbc0a8e00 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbbc1423b0 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc0b21f0 con 0x7fdbbc0a8a20 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc0a9ed0 con 0x7fdbbc1423b0 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbbc0a9cc0 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.405+0000 7fdbc4490640 10 monclient(hunting): authenticate will time out at 3436.735686s 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.406+0000 7fdbc1a2b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdbbc0a8e00 0x7fdbbc13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.406+0000 7fdbc2a2d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fdbbc1423b0 0x7fdbbc142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:52136/0 (socket says 172.21.15.86:52136) 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.406+0000 7fdbc2a2d640 1 -- 172.21.15.86:0/1646066151 learned_addr learned my addr 172.21.15.86:0/1646066151 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.406+0000 7fdbc1a2b640 10 monclient(hunting): get_auth_request con 0x7fdbbc0a8e00 auth_method 0 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.406+0000 7fdbc1a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.406+0000 7fdbc1a2b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.406+0000 7fdbc1a2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.407+0000 7fdbc1a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:10.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.407+0000 7fdbc1a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:10.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.407+0000 7fdbc1a2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:10.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc1a2b640 10 monclient(hunting): handle_auth_done global_id 16929 payload 274 2023-12-11T20:49:10.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc122a640 1 -- 172.21.15.86:0/1646066151 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217884425 0 0) 0x7fdbbc0a9ed0 con 0x7fdbbc1423b0 2023-12-11T20:49:10.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc1a2b640 1 -- 172.21.15.86:0/1646066151 >> v1:172.21.15.133:6789/0 conn(0x7fdbbc1423b0 legacy=0x7fdbbc142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc1a2b640 1 -- 172.21.15.86:0/1646066151 >> v1:172.21.15.86:6789/0 conn(0x7fdbbc0a8a20 legacy=0x7fdbbc13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:10.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc1a2b640 10 monclient: _finish_hunting 0 2023-12-11T20:49:10.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc1a2b640 1 monclient: found mon.noname-a 2023-12-11T20:49:10.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc1a2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:10.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc1a2b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:49:10.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc1a2b640 1 -- 172.21.15.86:0/1646066151 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0aac70 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.408+0000 7fdbc122a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-12-11T20:49:10.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.410+0000 7fdbc1a2b640 1 --2- 172.21.15.86:0/1646066151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 0x7fdbbc13f5c0 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0097a0 tx=0x7fdbac02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f013747b5a444ae server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 1 -- 172.21.15.86:0/1646066151 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac02f050 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: dump: 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: _finish_auth 0 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: _check_auth_tickets 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 1 -- 172.21.15.86:0/1646066151 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac031700 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 1 -- 172.21.15.86:0/1646066151 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac031a90 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc4490640 5 monclient: authenticate success, global_id 16929 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc4490640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc4490640 4 set_mon_vals no callback set 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:10.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.414+0000 7fdbc122a640 10 monclient: dump: 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:10.416 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 10 monclient: shutdown 2023-12-11T20:49:10.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:10.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 1 -- 172.21.15.86:0/1646066151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 msgr2=0x7fdbbc13f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 1 --2- 172.21.15.86:0/1646066151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 0x7fdbbc13f5c0 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0097a0 tx=0x7fdbac02b7a0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.419 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 1 -- 172.21.15.86:0/1646066151 shutdown_connections 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 1 --2- 172.21.15.86:0/1646066151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 0x7fdbbc13f5c0 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 1 -- 172.21.15.86:0/1646066151 >> 172.21.15.86:0/1646066151 conn(0x7fdbbc0b7870 msgr2=0x7fdbbc0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 1 -- 172.21.15.86:0/1646066151 shutdown_connections 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.415+0000 7fdbc4490640 1 -- 172.21.15.86:0/1646066151 wait complete. 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 10 monclient: build_initial_monmap 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 10 monclient: monmap: 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 1 librados: starting msgr at 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 1 librados: starting objecter 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 1 Processor -- start 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 1 -- start start 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 1 librados: setting wanted keys 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 1 librados: calling monclient init 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.416+0000 7fdbc4490640 10 monclient: init 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:10.421 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc0a8a20 0x7fdbbc145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient(hunting): picked mon.noname-b con 0x7fdbbc0a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 0x7fdbbc139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient(hunting): picked mon.noname-a con 0x7fdbbc0a8e00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbbc1423b0 0x7fdbbc0abb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient(hunting): picked mon.noname-c con 0x7fdbbc1423b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc13a5d0 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0ab6f0 con 0x7fdbbc0a8a20 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc13a1a0 con 0x7fdbbc1423b0 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:10.422 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc4490640 10 monclient(hunting): authenticate will time out at 3436.747254s 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc1a2b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 0x7fdbbc139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc2a2d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbbc1423b0 0x7fdbbc0abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.417+0000 7fdbc222c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc0a8a20 0x7fdbbc145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc0a8a20 0x7fdbbc145ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:35520/0 (socket says 172.21.15.86:35520) 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 1 -- 172.21.15.86:0/3345401633 learned_addr learned my addr 172.21.15.86:0/3345401633 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc2a2d640 10 monclient(hunting): get_auth_request con 0x7fdbbc1423b0 auth_method 0 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc2a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc2a2d640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc2a2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 10 monclient(hunting): get_auth_request con 0x7fdbbc0a8a20 auth_method 0 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc1a2b640 10 monclient(hunting): get_auth_request con 0x7fdbbc0a8e00 auth_method 0 2023-12-11T20:49:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc1a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc1a2b640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc1a2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc222c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc1a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc1a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc1a2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc2a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc2a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:10.424 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.418+0000 7fdbc2a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:10.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 10 monclient(hunting): handle_auth_done global_id 16932 payload 824 2023-12-11T20:49:10.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 1 -- 172.21.15.86:0/3345401633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbbc1423b0 msgr2=0x7fdbbc0abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.425 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbbc1423b0 0x7fdbbc0abb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 1 -- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc0a8a20 msgr2=0x7fdbbc145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc0a8a20 0x7fdbbc145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 10 monclient: _finish_hunting 0 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 1 monclient: found mon.noname-a 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.421+0000 7fdbc1a2b640 1 -- 172.21.15.86:0/3345401633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac009400 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.423+0000 7fdbc1a2b640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 0x7fdbbc139c60 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fdbac033000 tx=0x7fdbac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.423+0000 7fdbb2ffd640 10 client.?.objecter ms_handle_connect 0x7fdbbc0a8e00 2023-12-11T20:49:10.426 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.423+0000 7fdbb2ffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:10.427 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 1 -- 172.21.15.86:0/3345401633 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac033c70 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.427 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:10.427 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: dump: 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: _finish_auth 0 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:10.428 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 1 -- 172.21.15.86:0/3345401633 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac033df0 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 5 monclient: authenticate success, global_id 16932 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 10 monclient: _renew_subs 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc13aa60 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 1 -- 172.21.15.86:0/3345401633 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac03e6c0 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbb2ffd640 10 monclient: dump: 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc0a29640 4 set_mon_vals no callback set 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 10 client.16932.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 10 monclient: _renew_subs 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc0ac230 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.427+0000 7fdbc4490640 1 librados: init done 2023-12-11T20:49:10.431 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.428+0000 7fdbc4490640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:10.431 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.428+0000 7fdbc4490640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:10.431 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.428+0000 7fdbc4490640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2023-12-11T20:49:10.431 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.428+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbbc0ac4e0 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.446+0000 7fdbb2ffd640 1 -- 172.21.15.86:0/3345401633 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdbac03a040 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.446 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.447+0000 7fdbb2ffd640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fdba006c820 0x7fdba006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:10.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.447+0000 7fdbc222c640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fdba006c820 0x7fdba006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:10.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.447+0000 7fdbc222c640 10 monclient: get_auth_request con 0x7fdba006c820 auth_method 0 2023-12-11T20:49:10.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.448+0000 7fdbc222c640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fdba006c820 0x7fdba006ece0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80018b0 tx=0x7fdbb8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:10.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.448+0000 7fdbb2ffd640 10 client.16932.objecter ms_handle_connect 0x7fdba006c820 2023-12-11T20:49:10.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.684+0000 7fdbb2ffd640 1 -- 172.21.15.86:0/3345401633 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fdbac07a900 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.684+0000 7fdbb2ffd640 10 client.16932.objecter ms_dispatch 0x7fdbbc000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:10.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.684+0000 7fdbb2ffd640 3 client.16932.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:10.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.684+0000 7fdbb2ffd640 3 client.16932.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:10.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.685+0000 7fdbb2ffd640 20 client.16932.objecter dump_active .. 0 homeless 2023-12-11T20:49:10.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.685+0000 7fdbb2ffd640 1 -- 172.21.15.86:0/3345401633 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbac076c40 con 0x7fdbbc0a8e00 2023-12-11T20:49:10.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.685+0000 7fdbb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:10.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.685+0000 7fdbb2ffd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:10.870 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.871+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbbc0ab580 con 0x7fdba006c820 2023-12-11T20:49:10.871 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.871+0000 7fdbb2ffd640 1 -- 172.21.15.86:0/3345401633 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdbbc0ab580 con 0x7fdba006c820 2023-12-11T20:49:10.871 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 5 to deep-scrub 2023-12-11T20:49:10.873 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 10 librados: watch_flush enter 2023-12-11T20:49:10.873 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 10 librados: watch_flush exit 2023-12-11T20:49:10.873 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 20 client.16932.objecter shutdown clearing up homeless session... 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 10 client.16932.objecter successfully canceled tick 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fdba006c820 msgr2=0x7fdba006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fdba006c820 0x7fdba006ece0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80018b0 tx=0x7fdbb8009040 comp rx=0 tx=0).stop 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 10 monclient: shutdown 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 msgr2=0x7fdbbc139c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 0x7fdbbc139c60 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fdbac033000 tx=0x7fdbac0046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 shutdown_connections 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fdba006c820 0x7fdba006ece0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbbc1423b0 0x7fdbbc0abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbbc0a8e00 0x7fdbbc139c60 unknown :-1 s=CLOSED pgs=3360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 --2- 172.21.15.86:0/3345401633 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc0a8a20 0x7fdbbc145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 >> 172.21.15.86:0/3345401633 conn(0x7fdbbc0b7870 msgr2=0x7fdbbc0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:10.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 shutdown_connections 2023-12-11T20:49:10.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 -- 172.21.15.86:0/3345401633 wait complete. 2023-12-11T20:49:10.875 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:10.873+0000 7fdbc4490640 1 librados: shutdown 2023-12-11T20:49:10.884 INFO:tasks.ceph:Scrubbing osd.6 2023-12-11T20:49:10.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.007+0000 7f866fd61640 10 monclient: get_monmap_and_config 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.007+0000 7f866fd61640 10 monclient: build_initial_monmap 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.007+0000 7f866fd61640 10 monclient: monmap: 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:11.009052+0000 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:11.009052+0000 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:11.007 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.008 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:11.008 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:11.008 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:11.008 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:11.008 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:11.008 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 1 Processor -- start 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 1 -- start start 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient: init 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f866806a770 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8668072890 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86681426e0 0x7f866813f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient(hunting): picked mon.noname-a con 0x7f86681426e0 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:11.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86680ad570 con 0x7f866806a770 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86680aeb40 con 0x7f8668072890 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866806ab50 con 0x7f86681426e0 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.008+0000 7f866fd61640 10 monclient(hunting): authenticate will time out at 3437.339084s 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.009+0000 7f866e2fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86681426e0 0x7f866813f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.009+0000 7f866d2fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f8668072890 0x7f866813eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:52146/0 (socket says 172.21.15.86:52146) 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.009+0000 7f866d2fc640 1 -- 172.21.15.86:0/410300530 learned_addr learned my addr 172.21.15.86:0/410300530 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.009+0000 7f866e2fe640 10 monclient(hunting): get_auth_request con 0x7f86681426e0 auth_method 0 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.009+0000 7f866e2fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.009+0000 7f866e2fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.009+0000 7f866e2fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.010+0000 7f866e2fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.010+0000 7f866e2fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.010+0000 7f866e2fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.011+0000 7f866cafb640 1 -- 172.21.15.86:0/410300530 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565681964 0 0) 0x7f86680aeb40 con 0x7f8668072890 2023-12-11T20:49:11.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.011+0000 7f866cafb640 10 monclient(hunting): _negotiate 2023-12-11T20:49:11.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.011+0000 7f866cafb640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.012 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.011+0000 7f866cafb640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.011+0000 7f866cafb640 10 monclient(hunting): my global_id is 21088 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.011+0000 7f866cafb640 1 -- 172.21.15.86:0/410300530 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8654003520 con 0x7f8668072890 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.012+0000 7f866e2fe640 10 monclient(hunting): handle_auth_done global_id 16935 payload 274 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.012+0000 7f866e2fe640 1 -- 172.21.15.86:0/410300530 >> v1:172.21.15.133:6789/0 conn(0x7f8668072890 legacy=0x7f866813eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.012+0000 7f866e2fe640 1 -- 172.21.15.86:0/410300530 >> v1:172.21.15.86:6789/0 conn(0x7f866806a770 legacy=0x7f8668141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.012+0000 7f866e2fe640 10 monclient: _finish_hunting 0 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.012+0000 7f866e2fe640 1 monclient: found mon.noname-a 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.012+0000 7f866e2fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.012+0000 7f866e2fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:49:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.012+0000 7f866e2fe640 1 -- 172.21.15.86:0/410300530 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86680ad1f0 con 0x7f86681426e0 2023-12-11T20:49:11.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.014+0000 7f866e2fe640 1 --2- 172.21.15.86:0/410300530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86681426e0 0x7f866813f270 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f8660009d90 tx=0x7f866002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59d7c081601c063 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.021 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 1 -- 172.21.15.86:0/410300530 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866002cbd0 con 0x7f86681426e0 2023-12-11T20:49:11.021 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: dump: 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.022 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: _finish_auth 0 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: _check_auth_tickets 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 1 -- 172.21.15.86:0/410300530 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866002cd30 con 0x7f86681426e0 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866fd61640 5 monclient: authenticate success, global_id 16935 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866fd61640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866fd61640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 1 -- 172.21.15.86:0/410300530 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866002cea0 con 0x7f86681426e0 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866fd61640 4 set_mon_vals no callback set 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.021+0000 7f866cafb640 10 monclient: dump: 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.024 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 10 monclient: shutdown 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 1 -- 172.21.15.86:0/410300530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86681426e0 msgr2=0x7f866813f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 1 --2- 172.21.15.86:0/410300530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86681426e0 0x7f866813f270 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f8660009d90 tx=0x7f866002be20 comp rx=0 tx=0).stop 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 1 -- 172.21.15.86:0/410300530 shutdown_connections 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 1 --2- 172.21.15.86:0/410300530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86681426e0 0x7f866813f270 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 1 -- 172.21.15.86:0/410300530 >> 172.21.15.86:0/410300530 conn(0x7f86680b4ba0 msgr2=0x7f86680b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 1 -- 172.21.15.86:0/410300530 shutdown_connections 2023-12-11T20:49:11.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.022+0000 7f866fd61640 1 -- 172.21.15.86:0/410300530 wait complete. 2023-12-11T20:49:11.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 10 monclient: build_initial_monmap 2023-12-11T20:49:11.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 10 monclient: monmap: 2023-12-11T20:49:11.028 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:11.028 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:11.028 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:11.028 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:11.028 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 1 librados: starting msgr at 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 1 librados: starting objecter 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 1 Processor -- start 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 1 -- start start 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 1 librados: setting wanted keys 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 1 librados: calling monclient init 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.023+0000 7f866fd61640 10 monclient: init 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:11.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f866806a770 0x7f8668070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient(hunting): picked mon.noname-c con 0x7f866806a770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8668072890 0x7f86680b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient(hunting): picked mon.noname-b con 0x7f8668072890 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86681426e0 0x7f86681da2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient(hunting): picked mon.noname-a con 0x7f86681426e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8668071f90 con 0x7f86681426e0 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8668137a60 con 0x7f8668072890 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8668071a20 con 0x7f866806a770 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866fd61640 10 monclient(hunting): authenticate will time out at 3437.354378s 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866dafd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f866806a770 0x7f8668070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866d2fc640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8668072890 0x7f86680b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866dafd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f866806a770 0x7f8668070f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53134/0 (socket says 172.21.15.86:53134) 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866dafd640 1 -- 172.21.15.86:0/3460432978 learned_addr learned my addr 172.21.15.86:0/3460432978 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866dafd640 10 monclient(hunting): get_auth_request con 0x7f866806a770 auth_method 0 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866dafd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866dafd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866dafd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.024+0000 7f866e2fe640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86681426e0 0x7f86681da2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866d2fc640 10 monclient(hunting): get_auth_request con 0x7f8668072890 auth_method 0 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866d2fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866d2fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866d2fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866e2fe640 10 monclient(hunting): get_auth_request con 0x7f86681426e0 auth_method 0 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866e2fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866e2fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866e2fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866dafd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866dafd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866dafd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866d2fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866d2fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866d2fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866e2fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866e2fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.025+0000 7f866e2fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 10 monclient(hunting): handle_auth_done global_id 21091 payload 824 2023-12-11T20:49:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 1 -- 172.21.15.86:0/3460432978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f866806a770 msgr2=0x7f8668070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f866806a770 0x7f8668070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 1 -- 172.21.15.86:0/3460432978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86681426e0 msgr2=0x7f86681da2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86681426e0 0x7f86681da2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 10 monclient: _finish_hunting 0 2023-12-11T20:49:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 1 monclient: found mon.noname-b 2023-12-11T20:49:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.028+0000 7f866d2fc640 1 -- 172.21.15.86:0/3460432978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8660009a70 con 0x7f8668072890 2023-12-11T20:49:11.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.030+0000 7f866d2fc640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8668072890 0x7f86680b3f10 secure :-1 s=READY pgs=5127 cs=0 l=1 rev1=1 crypto rx=0x7f865800c400 tx=0x7f865800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.030+0000 7f865e7fc640 10 client.?.objecter ms_handle_connect 0x7f8668072890 2023-12-11T20:49:11.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.030+0000 7f865e7fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8658005ce0 con 0x7f8668072890 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: dump: 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: _finish_auth 0 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8658005e40 con 0x7f8668072890 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86580072d0 con 0x7f8668072890 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 5 monclient: authenticate success, global_id 21091 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 10 monclient: _renew_subs 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8668071ca0 con 0x7f8668072890 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865ffff640 4 set_mon_vals no callback set 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f865e7fc640 10 monclient: dump: 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.036 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 10 client.21091.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 10 monclient: _renew_subs 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86681da9f0 con 0x7f8668072890 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.033+0000 7f866fd61640 1 librados: init done 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.034+0000 7f866fd61640 10 client.21091.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:11.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.034+0000 7f866fd61640 15 client.21091.objecter _session_command_op_assign -1 1 2023-12-11T20:49:11.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.034+0000 7f866fd61640 20 client.21091.objecter _get_session osd=-1 returning homeless 2023-12-11T20:49:11.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.034+0000 7f866fd61640 10 client.21091.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:11.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.034+0000 7f866fd61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.034+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f86680ad9e0 con 0x7f8668072890 2023-12-11T20:49:11.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.053+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8658007430 con 0x7f8668072890 2023-12-11T20:49:11.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.053+0000 7f865e7fc640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f864c06c870 0x7f864c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.053+0000 7f866dafd640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f864c06c870 0x7f864c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.053+0000 7f866dafd640 10 monclient: get_auth_request con 0x7f864c06c870 auth_method 0 2023-12-11T20:49:11.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.054+0000 7f866dafd640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f864c06c870 0x7f864c06ed30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f86640018b0 tx=0x7f8664009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.053 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.054+0000 7f865e7fc640 10 client.21091.objecter ms_handle_connect 0x7f864c06c870 2023-12-11T20:49:11.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f86580584b0 con 0x7f8668072890 2023-12-11T20:49:11.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 10 client.21091.objecter ms_dispatch 0x7f8668000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 3 client.21091.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 3 client.21091.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 10 client.21091.objecter checking command 1 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7f864c074520 0x7f864c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 20 client.21091.objecter _get_session s=0x7f864c073e50 osd=6 3 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 20 client.21091.objecter put_session s=0x7f864c073e50 osd=6 3 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 15 client.21091.objecter _session_command_op_remove -1 1 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 20 client.21091.objecter _get_session s=0x7f864c073e50 osd=6 3 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 20 client.21091.objecter get_session s=0x7f864c073e50 osd=6 3 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 15 client.21091.objecter _session_command_op_assign 6 1 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 20 client.21091.objecter put_session s=0x7f864c073e50 osd=6 4 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 10 client.21091.objecter _send_command 1 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f864c077090 con 0x7f864c074520 2023-12-11T20:49:11.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 20 client.21091.objecter dump_active .. 0 homeless 2023-12-11T20:49:11.057 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f866e2fe640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7f864c074520 0x7f864c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.057 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86580547f0 con 0x7f8668072890 2023-12-11T20:49:11.057 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.056+0000 7f865e7fc640 10 monclient: handle_get_version_reply finishing 1 version 1057 2023-12-11T20:49:11.057 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.057+0000 7f866e2fe640 10 monclient: get_auth_request con 0x7f864c074520 auth_method 0 2023-12-11T20:49:11.059 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.060+0000 7f865e7fc640 10 client.21091.objecter ms_handle_connect 0x7f864c074520 2023-12-11T20:49:11.059 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.060+0000 7f866e2fe640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7f864c074520 0x7f864c0769c0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.084+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 <== osd.6 v2:172.21.15.133:6826/1611390427 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f864c077090 con 0x7f864c074520 2023-12-11T20:49:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.084+0000 7f865e7fc640 10 client.21091.objecter ms_dispatch 0x7f8668000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:49:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.084+0000 7f865e7fc640 10 client.21091.objecter _finish_command 1 = system:0 2023-12-11T20:49:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.084+0000 7f865e7fc640 20 client.21091.objecter put_session s=0x7f864c073e50 osd=6 4 2023-12-11T20:49:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.084+0000 7f865e7fc640 15 client.21091.objecter _session_command_op_remove 6 1 2023-12-11T20:49:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 10 client.21091.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-11T20:49:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 15 client.21091.objecter _session_command_op_assign -1 2 2023-12-11T20:49:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 20 client.21091.objecter _get_session s=0x7f864c073e50 osd=6 3 2023-12-11T20:49:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 20 client.21091.objecter put_session s=0x7f864c073e50 osd=6 3 2023-12-11T20:49:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 20 client.21091.objecter _get_session s=0x7f864c073e50 osd=6 3 2023-12-11T20:49:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 15 client.21091.objecter _session_command_op_remove -1 2 2023-12-11T20:49:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 20 client.21091.objecter get_session s=0x7f864c073e50 osd=6 3 2023-12-11T20:49:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 15 client.21091.objecter _session_command_op_assign 6 2 2023-12-11T20:49:11.097 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 20 client.21091.objecter put_session s=0x7f864c073e50 osd=6 4 2023-12-11T20:49:11.097 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 10 client.21091.objecter _send_command 2 2023-12-11T20:49:11.097 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.096+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 --> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8668072100 con 0x7f864c074520 2023-12-11T20:49:11.110 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f865e7fc640 1 -- 172.21.15.86:0/3460432978 <== osd.6 v2:172.21.15.133:6826/1611390427 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8668072100 con 0x7f864c074520 2023-12-11T20:49:11.110 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f865e7fc640 10 client.21091.objecter ms_dispatch 0x7f8668000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:49:11.110 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f865e7fc640 10 client.21091.objecter _finish_command 2 = system:0 2023-12-11T20:49:11.111 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f865e7fc640 20 client.21091.objecter put_session s=0x7f864c073e50 osd=6 4 2023-12-11T20:49:11.111 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f865e7fc640 15 client.21091.objecter _session_command_op_remove 6 2 2023-12-11T20:49:11.111 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-12-11T20:49:11.111 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-11T20:49:11.111 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-12-11T20:49:11.112 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 10 librados: watch_flush enter 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 10 librados: watch_flush exit 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 10 client.21091.objecter close_session for osd.6 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7f864c074520 msgr2=0x7f864c0769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6826/1611390427,v1:172.21.15.133:6827/1611390427] conn(0x7f864c074520 0x7f864c0769c0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 20 client.21091.objecter put_session s=0x7f864c073e50 osd=6 1 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 20 client.21091.objecter shutdown clearing up homeless session... 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 10 client.21091.objecter successfully canceled tick 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f864c06c870 msgr2=0x7f864c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f864c06c870 0x7f864c06ed30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f86640018b0 tx=0x7f8664009040 comp rx=0 tx=0).stop 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 10 monclient: shutdown 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8668072890 msgr2=0x7f86680b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 --2- 172.21.15.86:0/3460432978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8668072890 0x7f86680b3f10 secure :-1 s=READY pgs=5127 cs=0 l=1 rev1=1 crypto rx=0x7f865800c400 tx=0x7f865800c8c0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.113 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866e2fe640 1 -- 172.21.15.86:0/3460432978 reap_dead start 2023-12-11T20:49:11.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 shutdown_connections 2023-12-11T20:49:11.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 >> 172.21.15.86:0/3460432978 conn(0x7f86680b4ba0 msgr2=0x7f866813e830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:11.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 shutdown_connections 2023-12-11T20:49:11.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 -- 172.21.15.86:0/3460432978 wait complete. 2023-12-11T20:49:11.114 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.111+0000 7f866fd61640 1 librados: shutdown 2023-12-11T20:49:11.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-11T20:49:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.227+0000 7f4612d16640 10 monclient: get_monmap_and_config 2023-12-11T20:49:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.227+0000 7f4612d16640 10 monclient: build_initial_monmap 2023-12-11T20:49:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.227+0000 7f4612d16640 10 monclient: monmap: 2023-12-11T20:49:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:11.228624+0000 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:11.228624+0000 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:11.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.227+0000 7f4612d16640 1 Processor -- start 2023-12-11T20:49:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.227+0000 7f4612d16640 1 -- start start 2023-12-11T20:49:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.227+0000 7f4612d16640 10 monclient: init 2023-12-11T20:49:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-12-11T20:49:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c0a8a20 0x7f460c13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 10 monclient(hunting): picked mon.noname-b con 0x7f460c0a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f460c0a8e00 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f460c1423b0 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c0b21f0 con 0x7f460c1423b0 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c142f20 con 0x7f460c0a8e00 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c0a9cc0 con 0x7f460c0a8a20 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.228+0000 7f4612d16640 10 monclient(hunting): authenticate will time out at 3437.558727s 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.229+0000 7f4610ab2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c0a8a20 0x7f460c13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.229+0000 7f460bfff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f460c0a8e00 0x7f460c13f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:52152/0 (socket says 172.21.15.86:52152) 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.229+0000 7f460bfff640 1 -- 172.21.15.86:0/4039282361 learned_addr learned my addr 172.21.15.86:0/4039282361 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.229+0000 7f4610ab2640 10 monclient(hunting): get_auth_request con 0x7f460c0a8a20 auth_method 0 2023-12-11T20:49:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.229+0000 7f4610ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.229+0000 7f4610ab2640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.229+0000 7f4610ab2640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.230+0000 7f4610ab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.230+0000 7f4610ab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.230+0000 7f4610ab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f460b7fe640 1 -- 172.21.15.86:0/4039282361 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920055986 0 0) 0x7f460c142f20 con 0x7f460c0a8e00 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f460b7fe640 10 monclient(hunting): _negotiate 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f460b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f460b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f460b7fe640 10 monclient(hunting): my global_id is 21094 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f460b7fe640 1 -- 172.21.15.86:0/4039282361 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f4003520 con 0x7f460c0a8e00 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f4610ab2640 10 monclient(hunting): handle_auth_done global_id 19061 payload 274 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f4610ab2640 1 -- 172.21.15.86:0/4039282361 >> v1:172.21.15.133:6789/0 conn(0x7f460c0a8e00 legacy=0x7f460c13f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f4610ab2640 1 -- 172.21.15.86:0/4039282361 >> v1:172.21.15.86:6789/0 conn(0x7f460c1423b0 legacy=0x7f460c142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f4610ab2640 10 monclient: _finish_hunting 0 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f4610ab2640 1 monclient: found mon.noname-b 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f4610ab2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f4610ab2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.231+0000 7f4610ab2640 1 -- 172.21.15.86:0/4039282361 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0a9ed0 con 0x7f460c0a8a20 2023-12-11T20:49:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.233+0000 7f4610ab2640 1 --2- 172.21.15.86:0/4039282361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c0a8a20 0x7f460c13eeb0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0097d0 tx=0x7f45fc02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e497be8eba63bd4e server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 1 -- 172.21.15.86:0/4039282361 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc031770 con 0x7f460c0a8a20 2023-12-11T20:49:11.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.237 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: dump: 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: _finish_auth 0 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: _check_auth_tickets 2023-12-11T20:49:11.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 1 -- 172.21.15.86:0/4039282361 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc0318d0 con 0x7f460c0a8a20 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 1 -- 172.21.15.86:0/4039282361 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc031c60 con 0x7f460c0a8a20 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f460b7fe640 10 monclient: dump: 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.239 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f4612d16640 5 monclient: authenticate success, global_id 19061 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f4612d16640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.237+0000 7f4612d16640 4 set_mon_vals no callback set 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 10 monclient: shutdown 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 1 -- 172.21.15.86:0/4039282361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c0a8a20 msgr2=0x7f460c13eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 1 --2- 172.21.15.86:0/4039282361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c0a8a20 0x7f460c13eeb0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0097d0 tx=0x7f45fc02b810 comp rx=0 tx=0).stop 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 1 -- 172.21.15.86:0/4039282361 shutdown_connections 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 1 --2- 172.21.15.86:0/4039282361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c0a8a20 0x7f460c13eeb0 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 1 -- 172.21.15.86:0/4039282361 >> 172.21.15.86:0/4039282361 conn(0x7f460c0b7870 msgr2=0x7f460c0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 1 -- 172.21.15.86:0/4039282361 shutdown_connections 2023-12-11T20:49:11.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 1 -- 172.21.15.86:0/4039282361 wait complete. 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 10 monclient: build_initial_monmap 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.238+0000 7f4612d16640 10 monclient: monmap: 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.239+0000 7f4612d16640 1 librados: starting msgr at 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.239+0000 7f4612d16640 1 librados: starting objecter 2023-12-11T20:49:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.239+0000 7f4612d16640 1 Processor -- start 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.239+0000 7f4612d16640 1 -- start start 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.239+0000 7f4612d16640 1 librados: setting wanted keys 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.239+0000 7f4612d16640 1 librados: calling monclient init 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.239+0000 7f4612d16640 10 monclient: init 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f460c0a8a20 0x7f460c144550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient(hunting): picked mon.noname-b con 0x7f460c0a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f460c0a8e00 0x7f460c145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient(hunting): picked mon.noname-a con 0x7f460c0a8e00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c1423b0 0x7f460c1395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient(hunting): picked mon.noname-c con 0x7f460c1423b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c144a90 con 0x7f460c0a8e00 2023-12-11T20:49:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c138940 con 0x7f460c0a8a20 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c138bf0 con 0x7f460c1423b0 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4612d16640 10 monclient(hunting): authenticate will time out at 3437.570207s 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f46112b3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c1423b0 0x7f460c1395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4610ab2640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f460c0a8a20 0x7f460c144550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f460bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f460c0a8e00 0x7f460c145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f46112b3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c1423b0 0x7f460c1395d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53154/0 (socket says 172.21.15.86:53154) 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f46112b3640 1 -- 172.21.15.86:0/157622343 learned_addr learned my addr 172.21.15.86:0/157622343 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f46112b3640 10 monclient(hunting): get_auth_request con 0x7f460c1423b0 auth_method 0 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f46112b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f46112b3640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f46112b3640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4610ab2640 10 monclient(hunting): get_auth_request con 0x7f460c0a8a20 auth_method 0 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4610ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4610ab2640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.240+0000 7f4610ab2640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f460bfff640 10 monclient(hunting): get_auth_request con 0x7f460c0a8e00 auth_method 0 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f460bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f460bfff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f460bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f4610ab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f4610ab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f4610ab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f46112b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f46112b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f46112b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f460bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f460bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.241+0000 7f460bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 10 monclient(hunting): handle_auth_done global_id 21097 payload 824 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 1 -- 172.21.15.86:0/157622343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c1423b0 msgr2=0x7f460c1395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c1423b0 0x7f460c1395d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 1 -- 172.21.15.86:0/157622343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f460c0a8e00 msgr2=0x7f460c145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f460c0a8e00 0x7f460c145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 10 monclient: _finish_hunting 0 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 1 monclient: found mon.noname-b 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.244+0000 7f4610ab2640 1 -- 172.21.15.86:0/157622343 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc009400 con 0x7f460c0a8a20 2023-12-11T20:49:11.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.246+0000 7f4610ab2640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f460c0a8a20 0x7f460c144550 secure :-1 s=READY pgs=5129 cs=0 l=1 rev1=1 crypto rx=0x7f45fc033000 tx=0x7f45fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.246+0000 7f46097fa640 10 client.?.objecter ms_handle_connect 0x7f460c0a8a20 2023-12-11T20:49:11.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.246+0000 7f46097fa640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:11.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.249+0000 7f46097fa640 1 -- 172.21.15.86:0/157622343 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc032810 con 0x7f460c0a8a20 2023-12-11T20:49:11.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.249+0000 7f46097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 10 monclient: dump: 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 10 monclient: _finish_auth 0 2023-12-11T20:49:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 10 monclient: _check_auth_tickets 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 1 -- 172.21.15.86:0/157622343 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc032990 con 0x7f460c0a8a20 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 1 -- 172.21.15.86:0/157622343 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc032d40 con 0x7f460c0a8a20 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f46097fa640 10 monclient: dump: 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 5 monclient: authenticate success, global_id 21097 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 10 monclient: _renew_subs 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f460affd640 4 set_mon_vals no callback set 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c138f30 con 0x7f460c0a8a20 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 10 client.21097.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 10 monclient: _renew_subs 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c138500 con 0x7f460c0a8a20 2023-12-11T20:49:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.250+0000 7f4612d16640 1 librados: init done 2023-12-11T20:49:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.251+0000 7f4612d16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.251+0000 7f4612d16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.251+0000 7f4612d16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.251+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460c145eb0 con 0x7f460c0a8a20 2023-12-11T20:49:11.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.269+0000 7f46097fa640 1 -- 172.21.15.86:0/157622343 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f45fc04e440 con 0x7f460c0a8a20 2023-12-11T20:49:11.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.269+0000 7f46097fa640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f45f806c8a0 0x7f45f806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.269+0000 7f460bfff640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f45f806c8a0 0x7f45f806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.269+0000 7f460bfff640 10 monclient: get_auth_request con 0x7f45f806c8a0 auth_method 0 2023-12-11T20:49:11.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.270+0000 7f46097fa640 10 client.21097.objecter ms_handle_connect 0x7f45f806c8a0 2023-12-11T20:49:11.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.270+0000 7f460bfff640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f45f806c8a0 0x7f45f806ed60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f460c0b1610 tx=0x7f4600009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.271+0000 7f46097fa640 1 -- 172.21.15.86:0/157622343 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f45fc07a7d0 con 0x7f460c0a8a20 2023-12-11T20:49:11.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.271+0000 7f46097fa640 10 client.21097.objecter ms_dispatch 0x7f460c000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:11.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.271+0000 7f46097fa640 3 client.21097.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:11.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.271+0000 7f46097fa640 3 client.21097.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:11.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.271+0000 7f46097fa640 20 client.21097.objecter dump_active .. 0 homeless 2023-12-11T20:49:11.502 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.503+0000 7f46097fa640 1 -- 172.21.15.86:0/157622343 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45fc076b10 con 0x7f460c0a8a20 2023-12-11T20:49:11.503 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.503+0000 7f46097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:11.503 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.503+0000 7f46097fa640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:11.670 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.670+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f460c13fb50 con 0x7f45f806c8a0 2023-12-11T20:49:11.671 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.671+0000 7f46097fa640 1 -- 172.21.15.86:0/157622343 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4600001580 con 0x7f45f806c8a0 2023-12-11T20:49:11.671 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 6 to deep-scrub 2023-12-11T20:49:11.673 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 10 librados: watch_flush enter 2023-12-11T20:49:11.673 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 10 librados: watch_flush exit 2023-12-11T20:49:11.673 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 20 client.21097.objecter shutdown clearing up homeless session... 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 10 client.21097.objecter successfully canceled tick 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f45f806c8a0 msgr2=0x7f45f806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f45f806c8a0 0x7f45f806ed60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f460c0b1610 tx=0x7f4600009040 comp rx=0 tx=0).stop 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 10 monclient: shutdown 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f460c0a8a20 msgr2=0x7f460c144550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f460c0a8a20 0x7f460c144550 secure :-1 s=READY pgs=5129 cs=0 l=1 rev1=1 crypto rx=0x7f45fc033000 tx=0x7f45fc0046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 shutdown_connections 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f45f806c8a0 0x7f45f806ed60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f460c1423b0 0x7f460c1395d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f460c0a8e00 0x7f460c145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 --2- 172.21.15.86:0/157622343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f460c0a8a20 0x7f460c144550 unknown :-1 s=CLOSED pgs=5129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 >> 172.21.15.86:0/157622343 conn(0x7f460c0b7870 msgr2=0x7f460c13bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 shutdown_connections 2023-12-11T20:49:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 -- 172.21.15.86:0/157622343 wait complete. 2023-12-11T20:49:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.673+0000 7f4612d16640 1 librados: shutdown 2023-12-11T20:49:11.683 INFO:tasks.ceph:Scrubbing osd.7 2023-12-11T20:49:11.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-11T20:49:11.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.788+0000 7f5f2c7a4640 10 monclient: get_monmap_and_config 2023-12-11T20:49:11.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.788+0000 7f5f2c7a4640 10 monclient: build_initial_monmap 2023-12-11T20:49:11.787 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.788+0000 7f5f2c7a4640 10 monclient: monmap: 2023-12-11T20:49:11.787 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:11.787 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:11.789533+0000 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:11.789533+0000 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:11.788 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.790 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.788+0000 7f5f2c7a4640 1 Processor -- start 2023-12-11T20:49:11.790 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.788+0000 7f5f2c7a4640 1 -- start start 2023-12-11T20:49:11.790 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.788+0000 7f5f2c7a4640 10 monclient: init 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f240a8a20 0x7f5f240a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient(hunting): picked mon.noname-a con 0x7f5f240a8a20 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f24145ab0 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413f800 0x7f5f24145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient(hunting): picked mon.noname-b con 0x7f5f2413f800 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f24146430 con 0x7f5f24145ab0 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f24145620 con 0x7f5f240a8a20 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f240aace0 con 0x7f5f2413f800 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2c7a4640 10 monclient(hunting): authenticate will time out at 3438.119573s 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.789+0000 7f5f2ad41640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413f800 0x7f5f24145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2a540640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f240a8a20 0x7f5f240a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413f800 0x7f5f24145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53164/0 (socket says 172.21.15.86:53164) 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 1 -- 172.21.15.86:0/3302451961 learned_addr learned my addr 172.21.15.86:0/3302451961 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 10 monclient(hunting): get_auth_request con 0x7f5f2413f800 auth_method 0 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2a540640 10 monclient(hunting): get_auth_request con 0x7f5f240a8a20 auth_method 0 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2a540640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2a540640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2a540640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2a540640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2a540640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.790+0000 7f5f2a540640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_done global_id 19067 payload 274 2023-12-11T20:49:11.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 1 -- 172.21.15.86:0/3302451961 >> v2:172.21.15.86:3300/0 conn(0x7f5f240a8a20 msgr2=0x7f5f240a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 1 --2- 172.21.15.86:0/3302451961 >> v2:172.21.15.86:3300/0 conn(0x7f5f240a8a20 0x7f5f240a8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 1 -- 172.21.15.86:0/3302451961 >> v1:172.21.15.133:6789/0 conn(0x7f5f24145ab0 legacy=0x7f5f2413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 10 monclient: _finish_hunting 0 2023-12-11T20:49:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 1 monclient: found mon.noname-b 2023-12-11T20:49:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.792+0000 7f5f2ad41640 1 -- 172.21.15.86:0/3302451961 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f240aafc0 con 0x7f5f2413f800 2023-12-11T20:49:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.794+0000 7f5f2ad41640 1 --2- 172.21.15.86:0/3302451961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413f800 0x7f5f24145ef0 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009e50 tx=0x7f5f1c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8a4c443901306c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 1 -- 172.21.15.86:0/3302451961 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c02cb00 con 0x7f5f2413f800 2023-12-11T20:49:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: dump: 2023-12-11T20:49:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: _finish_auth 0 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: _check_auth_tickets 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 1 -- 172.21.15.86:0/3302451961 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c02cc60 con 0x7f5f2413f800 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:11.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 1 -- 172.21.15.86:0/3302451961 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0343f0 con 0x7f5f2413f800 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2c7a4640 5 monclient: authenticate success, global_id 19067 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2c7a4640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2c7a4640 4 set_mon_vals no callback set 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.797+0000 7f5f2953e640 10 monclient: dump: 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.800 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.800 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 10 monclient: shutdown 2023-12-11T20:49:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/3302451961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413f800 msgr2=0x7f5f24145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 1 --2- 172.21.15.86:0/3302451961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413f800 0x7f5f24145ef0 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009e50 tx=0x7f5f1c02bf60 comp rx=0 tx=0).stop 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/3302451961 shutdown_connections 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 1 --2- 172.21.15.86:0/3302451961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413f800 0x7f5f24145ef0 unknown :-1 s=CLOSED pgs=3343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 1 --2- 172.21.15.86:0/3302451961 >> v2:172.21.15.86:3300/0 conn(0x7f5f240a8a20 0x7f5f240a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/3302451961 >> 172.21.15.86:0/3302451961 conn(0x7f5f240b7890 msgr2=0x7f5f240b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/3302451961 shutdown_connections 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.798+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/3302451961 wait complete. 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 10 monclient: build_initial_monmap 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 10 monclient: monmap: 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 1 librados: starting msgr at 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 1 librados: starting objecter 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 1 Processor -- start 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 1 -- start start 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 1 librados: setting wanted keys 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 1 librados: calling monclient init 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.799+0000 7f5f2c7a4640 10 monclient: init 2023-12-11T20:49:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240a8a20 0x7f5f24137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient(hunting): picked mon.noname-a con 0x7f5f240a8a20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 0x7f5f241432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient(hunting): picked mon.noname-b con 0x7f5f2413f800 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 0x7f5f24143800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient(hunting): picked mon.noname-c con 0x7f5f24145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f241329c0 con 0x7f5f240a8a20 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f24138f60 con 0x7f5f2413f800 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f24139240 con 0x7f5f24145ab0 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2c7a4640 10 monclient(hunting): authenticate will time out at 3438.130574s 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2ad41640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 0x7f5f24143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f29d3f640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 0x7f5f241432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.800+0000 7f5f2a540640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240a8a20 0x7f5f24137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 0x7f5f24143800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53172/0 (socket says 172.21.15.86:53172) 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 1 -- 172.21.15.86:0/4156109180 learned_addr learned my addr 172.21.15.86:0/4156109180 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 10 monclient(hunting): get_auth_request con 0x7f5f24145ab0 auth_method 0 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f29d3f640 10 monclient(hunting): get_auth_request con 0x7f5f2413f800 auth_method 0 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f29d3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f29d3f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f29d3f640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2a540640 10 monclient(hunting): get_auth_request con 0x7f5f240a8a20 auth_method 0 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2a540640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2a540640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2a540640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f29d3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f29d3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f29d3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2a540640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2a540640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.801+0000 7f5f2a540640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 10 monclient(hunting): handle_auth_done global_id 21103 payload 824 2023-12-11T20:49:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 msgr2=0x7f5f24143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 0x7f5f24143800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240a8a20 msgr2=0x7f5f24137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240a8a20 0x7f5f24137130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 10 monclient: _finish_hunting 0 2023-12-11T20:49:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 1 monclient: found mon.noname-b 2023-12-11T20:49:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.804+0000 7f5f29d3f640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c009b30 con 0x7f5f2413f800 2023-12-11T20:49:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.806+0000 7f5f29d3f640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 0x7f5f241432c0 secure :-1 s=READY pgs=5131 cs=0 l=1 rev1=1 crypto rx=0x7f5f1400c3c0 tx=0x7f5f1400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.806+0000 7f5f1affd640 10 client.?.objecter ms_handle_connect 0x7f5f2413f800 2023-12-11T20:49:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.806+0000 7f5f1affd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f140052d0 con 0x7f5f2413f800 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: dump: 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: _finish_auth 0 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f14005450 con 0x7f5f2413f800 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 5 monclient: authenticate success, global_id 21103 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 10 monclient: _renew_subs 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14007260 con 0x7f5f2413f800 2023-12-11T20:49:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f24137db0 con 0x7f5f2413f800 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1bfff640 4 set_mon_vals no callback set 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f1affd640 10 monclient: dump: 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 10 client.21103.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 10 monclient: _renew_subs 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f24137670 con 0x7f5f2413f800 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.809+0000 7f5f2c7a4640 1 librados: init done 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f29d3f640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 msgr2=0x7f5f241432c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f29d3f640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 msgr2=0x7f5f241432c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f29d3f640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 0x7f5f241432c0 secure :-1 s=READY pgs=5131 cs=0 l=1 rev1=1 crypto rx=0x7f5f1400c3c0 tx=0x7f5f1400c880 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f29d3f640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 0x7f5f241432c0 secure :-1 s=READY pgs=5131 cs=0 l=1 rev1=1 crypto rx=0x7f5f1400c3c0 tx=0x7f5f1400c880 comp rx=0 tx=0).stop 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 msgr2=0x7f5f241432c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-11T20:49:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f2413f800 0x7f5f241432c0 unknown :-1 s=CLOSED pgs=5131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f080051f0 0x7f5f080075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient(hunting): picked mon.b con 0x7f5f080051f0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240a8a20 0x7f5f0800a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient(hunting): picked mon.a con 0x7f5f240a8a20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 0x7f5f0800cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient(hunting): picked mon.c con 0x7f5f24145ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0800d570 con 0x7f5f240a8a20 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0800d6b0 con 0x7f5f080051f0 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0800d820 con 0x7f5f24145ab0 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f1affd640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f2c7a4640 10 client.21103.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f2c7a4640 15 client.21103.objecter _session_command_op_assign -1 1 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f2c7a4640 20 client.21103.objecter _get_session osd=-1 returning homeless 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.810+0000 7f5f2c7a4640 10 client.21103.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f29d3f640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240a8a20 0x7f5f0800a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f2ad41640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 0x7f5f0800cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f2ad41640 10 monclient(hunting): get_auth_request con 0x7f5f24145ab0 auth_method 0 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f2ad41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f2ad41640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f2ad41640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f29d3f640 10 monclient(hunting): get_auth_request con 0x7f5f240a8a20 auth_method 0 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f29d3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f29d3f640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f29d3f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-11T20:49:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f29d3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f29d3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.811+0000 7f5f29d3f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:49:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.812+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.812+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.812+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 10 monclient(hunting): handle_auth_done global_id 21103 payload 293 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f080051f0 msgr2=0x7f5f080075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f080051f0 0x7f5f080075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240a8a20 msgr2=0x7f5f0800a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240a8a20 0x7f5f0800a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 10 monclient: _finish_hunting 0 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 1 monclient: found mon.c 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f5f0800db10 con 0x7f5f24145ab0 2023-12-11T20:49:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f2414c880 con 0x7f5f24145ab0 2023-12-11T20:49:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 10 monclient: _finish_auth 0 2023-12-11T20:49:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 10 monclient: _check_auth_tickets 2023-12-11T20:49:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.813+0000 7f5f2ad41640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.815+0000 7f5f1affd640 10 client.21103.objecter ms_handle_connect 0x7f5f24145ab0 2023-12-11T20:49:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.815+0000 7f5f2ad41640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 0x7f5f0800cb60 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c036040 tx=0x7f5f14007e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.815+0000 7f5f1affd640 10 client.21103.objecter resend_mon_ops 2023-12-11T20:49:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.815+0000 7f5f1affd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.815+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=2) v1 -- 0x7f5f0800de20 con 0x7f5f24145ab0 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.837+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c03f440 con 0x7f5f24145ab0 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.837+0000 7f5f1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.837+0000 7f5f1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.837+0000 7f5f1affd640 10 monclient: dump: 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:11.838 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.837+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0800de20 con 0x7f5f24145ab0 2023-12-11T20:49:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.837+0000 7f5f1affd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.837+0000 7f5f28d3d640 4 set_mon_vals no callback set 2023-12-11T20:49:11.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.838+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5f1c03f5a0 con 0x7f5f24145ab0 2023-12-11T20:49:11.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.838+0000 7f5f1affd640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5f2413f800 0x7f5f08077230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.839+0000 7f5f2a540640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5f2413f800 0x7f5f08077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.839+0000 7f5f2a540640 10 monclient: get_auth_request con 0x7f5f2413f800 auth_method 0 2023-12-11T20:49:11.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.839+0000 7f5f1affd640 10 client.21103.objecter ms_handle_connect 0x7f5f2413f800 2023-12-11T20:49:11.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.839+0000 7f5f2a540640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5f2413f800 0x7f5f08077230 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009b50 tx=0x7f5f200096f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.841 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.2 v2:172.21.15.86:3301/0 4 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f5f1c0798e0 con 0x7f5f24145ab0 2023-12-11T20:49:11.841 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 10 client.21103.objecter ms_dispatch 0x7f5f24000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:11.841 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 3 client.21103.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:11.841 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 3 client.21103.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 10 client.21103.objecter checking command 1 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f5f0807c970 0x7f5f0807ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 20 client.21103.objecter _get_session s=0x7f5f0807c2a0 osd=7 3 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 20 client.21103.objecter put_session s=0x7f5f0807c2a0 osd=7 3 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 15 client.21103.objecter _session_command_op_remove -1 1 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 20 client.21103.objecter _get_session s=0x7f5f0807c2a0 osd=7 3 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 20 client.21103.objecter get_session s=0x7f5f0807c2a0 osd=7 3 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 15 client.21103.objecter _session_command_op_assign 7 1 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 20 client.21103.objecter put_session s=0x7f5f0807c2a0 osd=7 4 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 10 client.21103.objecter _send_command 1 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f0807f4e0 con 0x7f5f0807c970 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 20 client.21103.objecter dump_active .. 0 homeless 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.2 v2:172.21.15.86:3301/0 5 ==== mon_get_version_reply(handle=1 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f1c075c20 con 0x7f5f24145ab0 2023-12-11T20:49:11.842 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 10 monclient: handle_get_version_reply finishing 1 version 1057 2023-12-11T20:49:11.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=2 version=1057) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f1c0796b0 con 0x7f5f24145ab0 2023-12-11T20:49:11.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.841+0000 7f5f1affd640 10 monclient: handle_get_version_reply finishing 2 version 1057 2023-12-11T20:49:11.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.842+0000 7f5f29d3f640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f5f0807c970 0x7f5f0807ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:11.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.842+0000 7f5f29d3f640 10 monclient: get_auth_request con 0x7f5f0807c970 auth_method 0 2023-12-11T20:49:11.845 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.845+0000 7f5f1affd640 10 client.21103.objecter ms_handle_connect 0x7f5f0807c970 2023-12-11T20:49:11.845 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.845+0000 7f5f29d3f640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f5f0807c970 0x7f5f0807ee10 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:11.869 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.869+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== osd.7 v2:172.21.15.133:6810/1136721322 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5f0807f4e0 con 0x7f5f0807c970 2023-12-11T20:49:11.869 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.869+0000 7f5f1affd640 10 client.21103.objecter ms_dispatch 0x7f5f24000b90 command_reply(tid 1: 0 ) v1 2023-12-11T20:49:11.869 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.869+0000 7f5f1affd640 10 client.21103.objecter _finish_command 1 = system:0 2023-12-11T20:49:11.869 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.869+0000 7f5f1affd640 20 client.21103.objecter put_session s=0x7f5f0807c2a0 osd=7 4 2023-12-11T20:49:11.869 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.869+0000 7f5f1affd640 15 client.21103.objecter _session_command_op_remove 7 1 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 10 client.21103.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 15 client.21103.objecter _session_command_op_assign -1 2 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 20 client.21103.objecter _get_session s=0x7f5f0807c2a0 osd=7 3 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 20 client.21103.objecter put_session s=0x7f5f0807c2a0 osd=7 3 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 20 client.21103.objecter _get_session s=0x7f5f0807c2a0 osd=7 3 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 15 client.21103.objecter _session_command_op_remove -1 2 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 20 client.21103.objecter get_session s=0x7f5f0807c2a0 osd=7 3 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 15 client.21103.objecter _session_command_op_assign 7 2 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 20 client.21103.objecter put_session s=0x7f5f0807c2a0 osd=7 4 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 10 client.21103.objecter _send_command 2 2023-12-11T20:49:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.882+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 --> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5f24137ff0 con 0x7f5f0807c970 2023-12-11T20:49:11.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f1affd640 1 -- 172.21.15.86:0/4156109180 <== osd.7 v2:172.21.15.133:6810/1136721322 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5f24137ff0 con 0x7f5f0807c970 2023-12-11T20:49:11.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f1affd640 10 client.21103.objecter ms_dispatch 0x7f5f24000b90 command_reply(tid 2: 0 ) v1 2023-12-11T20:49:11.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f1affd640 10 client.21103.objecter _finish_command 2 = system:0 2023-12-11T20:49:11.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f1affd640 20 client.21103.objecter put_session s=0x7f5f0807c2a0 osd=7 4 2023-12-11T20:49:11.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f1affd640 15 client.21103.objecter _session_command_op_remove 7 2 2023-12-11T20:49:11.896 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-12-11T20:49:11.896 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-11T20:49:11.896 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-12-11T20:49:11.897 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 10 librados: watch_flush enter 2023-12-11T20:49:11.897 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 10 librados: watch_flush exit 2023-12-11T20:49:11.897 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 10 client.21103.objecter close_session for osd.7 2023-12-11T20:49:11.897 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f5f0807c970 msgr2=0x7f5f0807ee10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.897 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6810/1136721322,v1:172.21.15.133:6811/1136721322] conn(0x7f5f0807c970 0x7f5f0807ee10 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.897 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 20 client.21103.objecter put_session s=0x7f5f0807c2a0 osd=7 1 2023-12-11T20:49:11.897 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 20 client.21103.objecter shutdown clearing up homeless session... 2023-12-11T20:49:11.897 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 10 client.21103.objecter successfully canceled tick 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5f2413f800 msgr2=0x7f5f08077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f5f2413f800 0x7f5f08077230 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009b50 tx=0x7f5f200096f0 comp rx=0 tx=0).stop 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 10 monclient: shutdown 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 msgr2=0x7f5f0800cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 --2- 172.21.15.86:0/4156109180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24145ab0 0x7f5f0800cb60 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c036040 tx=0x7f5f14007e50 comp rx=0 tx=0).stop 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2ad41640 1 -- 172.21.15.86:0/4156109180 reap_dead start 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 shutdown_connections 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 >> 172.21.15.86:0/4156109180 conn(0x7f5f240b7890 msgr2=0x7f5f2413e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 shutdown_connections 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 -- 172.21.15.86:0/4156109180 wait complete. 2023-12-11T20:49:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:11.896+0000 7f5f2c7a4640 1 librados: shutdown 2023-12-11T20:49:11.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-11T20:49:12.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.012+0000 7f7358397640 10 monclient: get_monmap_and_config 2023-12-11T20:49:12.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.012+0000 7f7358397640 10 monclient: build_initial_monmap 2023-12-11T20:49:12.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.012+0000 7f7358397640 10 monclient: monmap: 2023-12-11T20:49:12.011 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:12.011 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:12.013541+0000 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:12.013541+0000 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:12.012 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.012+0000 7f7358397640 1 Processor -- start 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.012+0000 7f7358397640 1 -- start start 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.012+0000 7f7358397640 10 monclient: init 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient(hunting): picked mon.noname-b con 0x7f73500a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73500a8e00 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f73501423b0 0x7f7350142810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient(hunting): picked mon.noname-c con 0x7f73501423b0 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73500b21f0 con 0x7f73500a8e00 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73500a9cc0 con 0x7f73501423b0 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73500a9ed0 con 0x7f73500a8a20 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.013+0000 7f7358397640 10 monclient(hunting): authenticate will time out at 3438.343583s 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f73501423b0 0x7f7350142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f73501423b0 0x7f7350142810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:35546/0 (socket says 172.21.15.86:35546) 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 1 -- 172.21.15.86:0/1102604914 learned_addr learned my addr 172.21.15.86:0/1102604914 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 10 monclient(hunting): get_auth_request con 0x7f73501423b0 auth_method 0 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.015 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356934640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.016 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.014+0000 7f7356133640 1 --2- 172.21.15.86:0/1102604914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.015+0000 7f7356133640 10 monclient(hunting): get_auth_request con 0x7f73500a8a20 auth_method 0 2023-12-11T20:49:12.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.015+0000 7f7356133640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.015+0000 7f7356133640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.015+0000 7f7356133640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.015+0000 7f7356133640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.015+0000 7f7356133640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.015+0000 7f7356133640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.017 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 10 monclient(hunting): handle_auth_done global_id 21106 payload 274 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 1 -- 172.21.15.86:0/1102604914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 msgr2=0x7f735013eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 1 --2- 172.21.15.86:0/1102604914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 1 -- 172.21.15.86:0/1102604914 >> v1:172.21.15.86:6789/0 conn(0x7f73500a8e00 legacy=0x7f735013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 10 monclient: _finish_hunting 0 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 1 monclient: found mon.noname-c 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.016+0000 7f7356934640 1 -- 172.21.15.86:0/1102604914 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73500aac70 con 0x7f73501423b0 2023-12-11T20:49:12.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.018+0000 7f7356934640 1 --2- 172.21.15.86:0/1102604914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 0x7f7350142810 secure :-1 s=READY pgs=5132 cs=0 l=1 rev1=1 crypto rx=0x7f7348009e50 tx=0x7f734802bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e91ea08d0f065104 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 1 -- 172.21.15.86:0/1102604914 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734802c950 con 0x7f73501423b0 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: dump: 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:12.024 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: _finish_auth 0 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: _check_auth_tickets 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 1 -- 172.21.15.86:0/1102604914 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734802cab0 con 0x7f73501423b0 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7358397640 5 monclient: authenticate success, global_id 21106 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7358397640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7358397640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 1 -- 172.21.15.86:0/1102604914 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734802ce40 con 0x7f73501423b0 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7358397640 4 set_mon_vals no callback set 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.024+0000 7f7355131640 10 monclient: dump: 2023-12-11T20:49:12.025 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:12.026 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 10 monclient: shutdown 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 1 -- 172.21.15.86:0/1102604914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 msgr2=0x7f7350142810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 1 --2- 172.21.15.86:0/1102604914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 0x7f7350142810 secure :-1 s=READY pgs=5132 cs=0 l=1 rev1=1 crypto rx=0x7f7348009e50 tx=0x7f734802bdb0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 1 -- 172.21.15.86:0/1102604914 shutdown_connections 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 1 --2- 172.21.15.86:0/1102604914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 0x7f7350142810 unknown :-1 s=CLOSED pgs=5132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 1 --2- 172.21.15.86:0/1102604914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 1 -- 172.21.15.86:0/1102604914 >> 172.21.15.86:0/1102604914 conn(0x7f73500b7870 msgr2=0x7f73500b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 1 -- 172.21.15.86:0/1102604914 shutdown_connections 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.025+0000 7f7358397640 1 -- 172.21.15.86:0/1102604914 wait complete. 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 10 monclient: build_initial_monmap 2023-12-11T20:49:12.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 10 monclient: monmap: 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 1 librados: starting msgr at 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 1 librados: starting objecter 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 1 Processor -- start 2023-12-11T20:49:12.031 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 1 -- start start 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 1 librados: setting wanted keys 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 1 librados: calling monclient init 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 10 monclient: init 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 10 monclient(hunting): picked mon.noname-c con 0x7f73500a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.026+0000 7f7358397640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73500a8e00 0x7f7350138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 10 monclient(hunting): picked mon.noname-a con 0x7f73500a8e00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 0x7f73501dac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 10 monclient(hunting): picked mon.noname-b con 0x7f73501423b0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735013a590 con 0x7f73500a8e00 2023-12-11T20:49:12.032 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73501da540 con 0x7f73501423b0 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73501da790 con 0x7f73500a8a20 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7358397640 10 monclient(hunting): authenticate will time out at 3438.357173s 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7355932640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73500a8e00 0x7f7350138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7356133640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7356133640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53224/0 (socket says 172.21.15.86:53224) 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7356133640 1 -- 172.21.15.86:0/64383576 learned_addr learned my addr 172.21.15.86:0/64383576 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7356133640 10 monclient(hunting): get_auth_request con 0x7f73500a8a20 auth_method 0 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7356133640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7356133640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.027+0000 7f7356133640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7355932640 10 monclient(hunting): get_auth_request con 0x7f73500a8e00 auth_method 0 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7355932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7355932640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7355932640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356934640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 0x7f73501dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356133640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356133640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356133640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356934640 10 monclient(hunting): get_auth_request con 0x7f73501423b0 auth_method 0 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356934640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356934640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356934640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.028+0000 7f7356934640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.029+0000 7f7356934640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.034 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.029+0000 7f7356934640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.029+0000 7f7355932640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.029+0000 7f7355932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.029+0000 7f7355932640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 10 monclient(hunting): handle_auth_done global_id 19076 payload 824 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 1 -- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 msgr2=0x7f73501dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 0x7f73501dac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 1 -- 172.21.15.86:0/64383576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73500a8e00 msgr2=0x7f7350138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73500a8e00 0x7f7350138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 10 monclient: _finish_hunting 0 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 1 monclient: found mon.noname-c 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.031+0000 7f7356133640 1 -- 172.21.15.86:0/64383576 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7348009b30 con 0x7f73500a8a20 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.033+0000 7f7356133640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013ac70 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7f734c00e9b0 tx=0x7f734c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.033+0000 7f7346ffd640 10 client.?.objecter ms_handle_connect 0x7f73500a8a20 2023-12-11T20:49:12.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.033+0000 7f7346ffd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:12.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 1 -- 172.21.15.86:0/64383576 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c009670 con 0x7f73500a8a20 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: dump: 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:12.039 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: _finish_auth 0 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 1 -- 172.21.15.86:0/64383576 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c00c9a0 con 0x7f73500a8a20 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 5 monclient: authenticate success, global_id 19076 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 10 monclient: _renew_subs 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7350146140 con 0x7f73500a8a20 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 1 -- 172.21.15.86:0/64383576 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c00cd50 con 0x7f73500a8a20 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7346ffd640 10 monclient: dump: 2023-12-11T20:49:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7354930640 4 set_mon_vals no callback set 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 10 client.19076.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 10 monclient: _renew_subs 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73501db3b0 con 0x7f73500a8a20 2023-12-11T20:49:12.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.038+0000 7f7358397640 1 librados: init done 2023-12-11T20:49:12.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.039+0000 7f7358397640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:12.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.039+0000 7f7358397640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:12.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.039+0000 7f7358397640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:12.042 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.039+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73501da950 con 0x7f73500a8a20 2023-12-11T20:49:12.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.280+0000 7f7346ffd640 1 -- 172.21.15.86:0/64383576 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f734c013470 con 0x7f73500a8a20 2023-12-11T20:49:12.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.280+0000 7f7346ffd640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c8a0 0x7f733406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.280+0000 7f7355932640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c8a0 0x7f733406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.280+0000 7f7355932640 10 monclient: get_auth_request con 0x7f733406c8a0 auth_method 0 2023-12-11T20:49:12.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.281+0000 7f7346ffd640 10 client.19076.objecter ms_handle_connect 0x7f733406c8a0 2023-12-11T20:49:12.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.281+0000 7f7355932640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c8a0 0x7f733406ed60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f73500b4010 tx=0x7f7340009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.294+0000 7f7346ffd640 1 -- 172.21.15.86:0/64383576 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f734c058d60 con 0x7f73500a8a20 2023-12-11T20:49:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.294+0000 7f7346ffd640 10 client.19076.objecter ms_dispatch 0x7f7350000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.294+0000 7f7346ffd640 3 client.19076.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.294+0000 7f7346ffd640 3 client.19076.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.294+0000 7f7346ffd640 20 client.19076.objecter dump_active .. 0 homeless 2023-12-11T20:49:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.294+0000 7f7346ffd640 1 -- 172.21.15.86:0/64383576 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f734c0550a0 con 0x7f73500a8a20 2023-12-11T20:49:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.294+0000 7f7346ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.294+0000 7f7346ffd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:12.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.486+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f735013fb50 con 0x7f733406c8a0 2023-12-11T20:49:12.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.487+0000 7f7346ffd640 1 -- 172.21.15.86:0/64383576 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7340001440 con 0x7f733406c8a0 2023-12-11T20:49:12.486 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 7 to deep-scrub 2023-12-11T20:49:12.489 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 10 librados: watch_flush enter 2023-12-11T20:49:12.489 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 10 librados: watch_flush exit 2023-12-11T20:49:12.489 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 20 client.19076.objecter shutdown clearing up homeless session... 2023-12-11T20:49:12.489 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 10 client.19076.objecter successfully canceled tick 2023-12-11T20:49:12.489 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c8a0 msgr2=0x7f733406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:12.489 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c8a0 0x7f733406ed60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f73500b4010 tx=0x7f7340009040 comp rx=0 tx=0).stop 2023-12-11T20:49:12.489 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 10 monclient: shutdown 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 msgr2=0x7f735013ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013ac70 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7f734c00e9b0 tx=0x7f734c00ee70 comp rx=0 tx=0).stop 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 shutdown_connections 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f733406c8a0 0x7f733406ed60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73501423b0 0x7f73501dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73500a8e00 0x7f7350138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 --2- 172.21.15.86:0/64383576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73500a8a20 0x7f735013ac70 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 >> 172.21.15.86:0/64383576 conn(0x7f73500b7870 msgr2=0x7f73500b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 shutdown_connections 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 -- 172.21.15.86:0/64383576 wait complete. 2023-12-11T20:49:12.490 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.489+0000 7f7358397640 1 librados: shutdown 2023-12-11T20:49:12.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-11T20:49:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.601+0000 7f0f3069b640 10 monclient: get_monmap_and_config 2023-12-11T20:49:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.601+0000 7f0f3069b640 10 monclient: build_initial_monmap 2023-12-11T20:49:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.601+0000 7f0f3069b640 10 monclient: monmap: 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:12.602618+0000 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:12.602618+0000 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:12.601 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.601+0000 7f0f3069b640 1 Processor -- start 2023-12-11T20:49:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.601+0000 7f0f3069b640 1 -- start start 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.601+0000 7f0f3069b640 10 monclient: init 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f280a8a20 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f0f280a8e00 0x7f0f2813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient(hunting): picked mon.noname-c con 0x7f0f280a8e00 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f281423b0 0x7f0f28142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f281423b0 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f280b21f0 con 0x7f0f280a8a20 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f280a9cc0 con 0x7f0f281423b0 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f280a9ed0 con 0x7f0f280a8e00 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.602+0000 7f0f3069b640 10 monclient(hunting): authenticate will time out at 3438.932629s 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2ec38640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f281423b0 0x7f0f28142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f0f280a8e00 0x7f0f2813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f0f280a8e00 0x7f0f2813f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:35568/0 (socket says 172.21.15.86:35568) 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 1 -- 172.21.15.86:0/2486890065 learned_addr learned my addr 172.21.15.86:0/2486890065 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 10 monclient(hunting): get_auth_request con 0x7f0f280a8e00 auth_method 0 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2ec38640 10 monclient(hunting): get_auth_request con 0x7f0f281423b0 auth_method 0 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2ec38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2ec38640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2ec38640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.603+0000 7f0f2dc36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.604+0000 7f0f2ec38640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.604+0000 7f0f2ec38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.604+0000 7f0f2ec38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 10 monclient(hunting): handle_auth_done global_id 21112 payload 274 2023-12-11T20:49:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 1 -- 172.21.15.86:0/2486890065 >> v2:172.21.15.86:3300/0 conn(0x7f0f281423b0 msgr2=0x7f0f28142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 1 --2- 172.21.15.86:0/2486890065 >> v2:172.21.15.86:3300/0 conn(0x7f0f281423b0 0x7f0f28142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 1 -- 172.21.15.86:0/2486890065 >> v1:172.21.15.133:6789/0 conn(0x7f0f280a8a20 legacy=0x7f0f2813eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 10 monclient: _finish_hunting 0 2023-12-11T20:49:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 1 monclient: found mon.noname-c 2023-12-11T20:49:12.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:12.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.133:3300/0 2023-12-11T20:49:12.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.605+0000 7f0f2dc36640 1 -- 172.21.15.86:0/2486890065 --> v2:172.21.15.133:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f280aac70 con 0x7f0f280a8e00 2023-12-11T20:49:12.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.608+0000 7f0f2dc36640 1 --2- 172.21.15.86:0/2486890065 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813f5c0 secure :-1 s=READY pgs=5133 cs=0 l=1 rev1=1 crypto rx=0x7f0f280b5370 tx=0x7f0f1802b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a145543218229f11 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 1 -- 172.21.15.86:0/2486890065 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18031750 con 0x7f0f280a8e00 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: dump: 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: _finish_auth 0 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: _check_auth_tickets 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f3069b640 5 monclient: authenticate success, global_id 21112 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 1 -- 172.21.15.86:0/2486890065 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f180318b0 con 0x7f0f280a8e00 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f3069b640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f3069b640 4 set_mon_vals no callback set 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 1 -- 172.21.15.86:0/2486890065 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18031c40 con 0x7f0f280a8e00 2023-12-11T20:49:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.611+0000 7f0f2d435640 10 monclient: dump: 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:12.613 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 10 monclient: shutdown 2023-12-11T20:49:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 1 -- 172.21.15.86:0/2486890065 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 msgr2=0x7f0f2813f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 1 --2- 172.21.15.86:0/2486890065 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813f5c0 secure :-1 s=READY pgs=5133 cs=0 l=1 rev1=1 crypto rx=0x7f0f280b5370 tx=0x7f0f1802b7f0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 1 -- 172.21.15.86:0/2486890065 shutdown_connections 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 1 --2- 172.21.15.86:0/2486890065 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813f5c0 unknown :-1 s=CLOSED pgs=5133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 1 --2- 172.21.15.86:0/2486890065 >> v2:172.21.15.86:3300/0 conn(0x7f0f281423b0 0x7f0f28142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 1 -- 172.21.15.86:0/2486890065 >> 172.21.15.86:0/2486890065 conn(0x7f0f280b7870 msgr2=0x7f0f280b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 1 -- 172.21.15.86:0/2486890065 shutdown_connections 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.612+0000 7f0f3069b640 1 -- 172.21.15.86:0/2486890065 wait complete. 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 10 monclient: build_initial_monmap 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 10 monclient: monmap: 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 1 librados: starting msgr at 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 1 librados: starting objecter 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 1 Processor -- start 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 1 -- start start 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 1 librados: setting wanted keys 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 1 librados: calling monclient init 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 10 monclient: init 2023-12-11T20:49:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a8a20 0x7f0f28145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 10 monclient(hunting): picked mon.noname-c con 0x7f0f280a8a20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.613+0000 7f0f3069b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 10 monclient(hunting): picked mon.noname-b con 0x7f0f280a8e00 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f281423b0 0x7f0f28138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f281423b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2813a8a0 con 0x7f0f281423b0 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f281da620 con 0x7f0f280a8e00 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f281394e0 con 0x7f0f280a8a20 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f3069b640 10 monclient(hunting): authenticate will time out at 3438.944145s 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2dc36640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2ec38640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f281423b0 0x7f0f28138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2e437640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a8a20 0x7f0f28145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2dc36640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:35572/0 (socket says 172.21.15.86:35572) 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2dc36640 1 -- 172.21.15.86:0/632047801 learned_addr learned my addr 172.21.15.86:0/632047801 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2dc36640 10 monclient(hunting): get_auth_request con 0x7f0f280a8e00 auth_method 0 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2dc36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2dc36640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2dc36640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2ec38640 10 monclient(hunting): get_auth_request con 0x7f0f281423b0 auth_method 0 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2ec38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2ec38640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2ec38640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2e437640 10 monclient(hunting): get_auth_request con 0x7f0f280a8a20 auth_method 0 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2e437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2e437640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.614+0000 7f0f2e437640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2dc36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2dc36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2dc36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2ec38640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2ec38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2ec38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2e437640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2e437640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:12.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.615+0000 7f0f2e437640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:12.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 10 monclient(hunting): handle_auth_done global_id 21115 payload 824 2023-12-11T20:49:12.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 1 -- 172.21.15.86:0/632047801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a8a20 msgr2=0x7f0f28145f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:12.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a8a20 0x7f0f28145f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 1 -- 172.21.15.86:0/632047801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f281423b0 msgr2=0x7f0f28138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:12.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f281423b0 0x7f0f28138fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 10 monclient: _finish_hunting 0 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 1 monclient: found mon.noname-b 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.617+0000 7f0f2dc36640 1 -- 172.21.15.86:0/632047801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180094d0 con 0x7f0f280a8e00 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.619+0000 7f0f2dc36640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813ac50 secure :-1 s=READY pgs=5134 cs=0 l=1 rev1=1 crypto rx=0x7f0f180090b0 tx=0x7f0f180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.619+0000 7f0f1effd640 10 client.?.objecter ms_handle_connect 0x7f0f280a8e00 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.619+0000 7f0f1effd640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 1 -- 172.21.15.86:0/632047801 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18033a10 con 0x7f0f280a8e00 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:12.624 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: dump: 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: _finish_auth 0 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:12.625 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 1 -- 172.21.15.86:0/632047801 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f18033b70 con 0x7f0f280a8e00 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 5 monclient: authenticate success, global_id 21115 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 10 monclient: _renew_subs 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f281396a0 con 0x7f0f280a8e00 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 1 -- 172.21.15.86:0/632047801 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1803c400 con 0x7f0f280a8e00 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f1effd640 10 monclient: dump: 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:12.626 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f2cc34640 4 set_mon_vals no callback set 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 10 client.21115.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 10 monclient: _renew_subs 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f281dad20 con 0x7f0f280a8e00 2023-12-11T20:49:12.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.623+0000 7f0f3069b640 1 librados: init done 2023-12-11T20:49:12.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.624+0000 7f0f3069b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:12.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.624+0000 7f0f3069b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:12.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.624+0000 7f0f3069b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:12.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.624+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2804e840 con 0x7f0f280a8e00 2023-12-11T20:49:12.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.642+0000 7f0f1effd640 1 -- 172.21.15.86:0/632047801 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0f1803a040 con 0x7f0f280a8e00 2023-12-11T20:49:12.642 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.642+0000 7f0f1effd640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0f0c06c870 0x7f0f0c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:12.642 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.643+0000 7f0f2e437640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0f0c06c870 0x7f0f0c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:12.642 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.643+0000 7f0f2e437640 10 monclient: get_auth_request con 0x7f0f0c06c870 auth_method 0 2023-12-11T20:49:12.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.643+0000 7f0f1effd640 10 client.21115.objecter ms_handle_connect 0x7f0f0c06c870 2023-12-11T20:49:12.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.643+0000 7f0f2e437640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0f0c06c870 0x7f0f0c06ed30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009fd0 tx=0x7f0f2400d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:12.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.647+0000 7f0f1effd640 1 -- 172.21.15.86:0/632047801 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f0f1807a1d0 con 0x7f0f280a8e00 2023-12-11T20:49:12.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.647+0000 7f0f1effd640 10 client.21115.objecter ms_dispatch 0x7f0f28000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:12.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.647+0000 7f0f1effd640 3 client.21115.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:12.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.647+0000 7f0f1effd640 3 client.21115.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:12.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.647+0000 7f0f1effd640 20 client.21115.objecter dump_active .. 0 homeless 2023-12-11T20:49:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.874+0000 7f0f1effd640 1 -- 172.21.15.86:0/632047801 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f18076510 con 0x7f0f280a8e00 2023-12-11T20:49:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.874+0000 7f0f1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:12.874+0000 7f0f1effd640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.046+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f2813fb50 con 0x7f0f0c06c870 2023-12-11T20:49:13.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.051+0000 7f0f1effd640 1 -- 172.21.15.86:0/632047801 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150712 (secure 0 0 0) 0x7f0f24013340 con 0x7f0f0c06c870 2023-12-11T20:49:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:49:13.053 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 10 librados: watch_flush enter 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 10 librados: watch_flush exit 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 20 client.21115.objecter shutdown clearing up homeless session... 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 10 client.21115.objecter successfully canceled tick 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0f0c06c870 msgr2=0x7f0f0c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0f0c06c870 0x7f0f0c06ed30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009fd0 tx=0x7f0f2400d040 comp rx=0 tx=0).stop 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 10 monclient: shutdown 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 msgr2=0x7f0f2813ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.053+0000 7f0f3069b640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813ac50 secure :-1 s=READY pgs=5134 cs=0 l=1 rev1=1 crypto rx=0x7f0f180090b0 tx=0x7f0f180046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 shutdown_connections 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f0f0c06c870 0x7f0f0c06ed30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f281423b0 0x7f0f28138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f280a8e00 0x7f0f2813ac50 unknown :-1 s=CLOSED pgs=5134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 --2- 172.21.15.86:0/632047801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a8a20 0x7f0f28145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 >> 172.21.15.86:0/632047801 conn(0x7f0f280b7870 msgr2=0x7f0f280b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 shutdown_connections 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 -- 172.21.15.86:0/632047801 wait complete. 2023-12-11T20:49:13.056 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:13.054+0000 7f0f3069b640 1 librados: shutdown 2023-12-11T20:49:13.064 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":2034,"stamp":"2023-12-11T20:49:11.111227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":534620,"kb_used_data":17208,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754440100,"statfs":{"total":773094113280,"available":772546662400,"internally_reserved":0,"allocated":17620992,"data_stored":5245138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":529714737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001926"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1011,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:07.140638+0000","last_change":"2023-12-11T20:49:07.141741+0000","last_active":"2023-12-11T20:49:07.140638+0000","last_peered":"2023-12-11T20:49:07.140638+0000","last_clean":"2023-12-11T20:49:07.140638+0000","last_became_active":"2023-12-11T20:18:00.530681+0000","last_became_peered":"2023-12-11T20:18:00.530681+0000","last_unstale":"2023-12-11T20:49:07.140638+0000","last_undegraded":"2023-12-11T20:49:07.140638+0000","last_fullsized":"2023-12-11T20:49:07.140638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:07.138431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:07.138431+0000","last_clean_scrub_stamp":"2023-12-11T20:49:07.138431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:43:34.623842+0000","scrub_duration":0.031828385000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1010,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.142477+0000","last_change":"2023-12-11T20:18:01.470212+0000","last_active":"2023-12-11T20:49:02.142477+0000","last_peered":"2023-12-11T20:49:02.142477+0000","last_clean":"2023-12-11T20:49:02.142477+0000","last_became_active":"2023-12-11T20:18:01.459740+0000","last_became_peered":"2023-12-11T20:18:01.459740+0000","last_unstale":"2023-12-11T20:49:02.142477+0000","last_undegraded":"2023-12-11T20:49:02.142477+0000","last_fullsized":"2023-12-11T20:49:02.142477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1011,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.877928+0000","last_change":"2023-12-11T20:18:11.560096+0000","last_active":"2023-12-11T20:49:00.877928+0000","last_peered":"2023-12-11T20:49:00.877928+0000","last_clean":"2023-12-11T20:49:00.877928+0000","last_became_active":"2023-12-11T20:18:11.552993+0000","last_became_peered":"2023-12-11T20:18:11.552993+0000","last_unstale":"2023-12-11T20:49:00.877928+0000","last_undegraded":"2023-12-11T20:49:00.877928+0000","last_fullsized":"2023-12-11T20:49:00.877928+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:31:04.664827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1007,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.297612+0000","last_change":"2023-12-11T20:18:04.754752+0000","last_active":"2023-12-11T20:49:02.297612+0000","last_peered":"2023-12-11T20:49:02.297612+0000","last_clean":"2023-12-11T20:49:02.297612+0000","last_became_active":"2023-12-11T20:18:04.748213+0000","last_became_peered":"2023-12-11T20:18:04.748213+0000","last_unstale":"2023-12-11T20:49:02.297612+0000","last_undegraded":"2023-12-11T20:49:02.297612+0000","last_fullsized":"2023-12-11T20:49:02.297612+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1010,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.305235+0000","last_change":"2023-12-11T20:18:07.107260+0000","last_active":"2023-12-11T20:49:02.305235+0000","last_peered":"2023-12-11T20:49:02.305235+0000","last_clean":"2023-12-11T20:49:02.305235+0000","last_became_active":"2023-12-11T20:18:06.871380+0000","last_became_peered":"2023-12-11T20:18:06.871380+0000","last_unstale":"2023-12-11T20:49:02.305235+0000","last_undegraded":"2023-12-11T20:49:02.305235+0000","last_fullsized":"2023-12-11T20:49:02.305235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1012,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.139421+0000","last_change":"2023-12-11T20:18:00.913863+0000","last_active":"2023-12-11T20:49:02.139421+0000","last_peered":"2023-12-11T20:49:02.139421+0000","last_clean":"2023-12-11T20:49:02.139421+0000","last_became_active":"2023-12-11T20:18:00.891399+0000","last_became_peered":"2023-12-11T20:18:00.891399+0000","last_unstale":"2023-12-11T20:49:02.139421+0000","last_undegraded":"2023-12-11T20:49:02.139421+0000","last_fullsized":"2023-12-11T20:49:02.139421+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1013,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.892004+0000","last_change":"2023-12-11T20:18:11.329207+0000","last_active":"2023-12-11T20:49:00.892004+0000","last_peered":"2023-12-11T20:49:00.892004+0000","last_clean":"2023-12-11T20:49:00.892004+0000","last_became_active":"2023-12-11T20:18:11.323743+0000","last_became_peered":"2023-12-11T20:18:11.323743+0000","last_unstale":"2023-12-11T20:49:00.892004+0000","last_undegraded":"2023-12-11T20:49:00.892004+0000","last_fullsized":"2023-12-11T20:49:00.892004+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:45:34.411191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1024,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.864920+0000","last_change":"2023-12-11T20:18:03.803423+0000","last_active":"2023-12-11T20:49:00.864920+0000","last_peered":"2023-12-11T20:49:00.864920+0000","last_clean":"2023-12-11T20:49:00.864920+0000","last_became_active":"2023-12-11T20:18:03.798156+0000","last_became_peered":"2023-12-11T20:18:03.798156+0000","last_unstale":"2023-12-11T20:49:00.864920+0000","last_undegraded":"2023-12-11T20:49:00.864920+0000","last_fullsized":"2023-12-11T20:49:00.864920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:31:17.842399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1017,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.957818+0000","last_change":"2023-12-11T20:18:06.556522+0000","last_active":"2023-12-11T20:49:00.957818+0000","last_peered":"2023-12-11T20:49:00.957818+0000","last_clean":"2023-12-11T20:49:00.957818+0000","last_became_active":"2023-12-11T20:18:06.509738+0000","last_became_peered":"2023-12-11T20:18:06.509738+0000","last_unstale":"2023-12-11T20:49:00.957818+0000","last_undegraded":"2023-12-11T20:49:00.957818+0000","last_fullsized":"2023-12-11T20:49:00.957818+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T00:20:54.755785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"118'7","reported_seq":1026,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.874651+0000","last_change":"2023-12-11T20:18:05.708954+0000","last_active":"2023-12-11T20:49:00.874651+0000","last_peered":"2023-12-11T20:49:00.874651+0000","last_clean":"2023-12-11T20:49:00.874651+0000","last_became_active":"2023-12-11T20:18:05.684774+0000","last_became_peered":"2023-12-11T20:18:05.684774+0000","last_unstale":"2023-12-11T20:49:00.874651+0000","last_undegraded":"2023-12-11T20:49:00.874651+0000","last_fullsized":"2023-12-11T20:49:00.874651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:35:07.384658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1010,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.292060+0000","last_change":"2023-12-11T20:18:11.631063+0000","last_active":"2023-12-11T20:49:02.292060+0000","last_peered":"2023-12-11T20:49:02.292060+0000","last_clean":"2023-12-11T20:49:02.292060+0000","last_became_active":"2023-12-11T20:18:11.576567+0000","last_became_peered":"2023-12-11T20:18:11.576567+0000","last_unstale":"2023-12-11T20:49:02.292060+0000","last_undegraded":"2023-12-11T20:49:02.292060+0000","last_fullsized":"2023-12-11T20:49:02.292060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1024,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.856011+0000","last_change":"2023-12-11T20:18:03.266276+0000","last_active":"2023-12-11T20:49:00.856011+0000","last_peered":"2023-12-11T20:49:00.856011+0000","last_clean":"2023-12-11T20:49:00.856011+0000","last_became_active":"2023-12-11T20:18:03.237738+0000","last_became_peered":"2023-12-11T20:18:03.237738+0000","last_unstale":"2023-12-11T20:49:00.856011+0000","last_undegraded":"2023-12-11T20:49:00.856011+0000","last_fullsized":"2023-12-11T20:49:00.856011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:00:38.864099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1057'78","reported_seq":1133,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:08.936824+0000","last_change":"2023-12-11T20:49:08.937802+0000","last_active":"2023-12-11T20:49:08.936824+0000","last_peered":"2023-12-11T20:49:08.936824+0000","last_clean":"2023-12-11T20:49:08.936824+0000","last_became_active":"2023-12-11T20:16:49.435576+0000","last_became_peered":"2023-12-11T20:16:49.435576+0000","last_unstale":"2023-12-11T20:49:08.936824+0000","last_undegraded":"2023-12-11T20:49:08.936824+0000","last_fullsized":"2023-12-11T20:49:08.936824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"1057'78","last_scrub_stamp":"2023-12-11T20:49:08.935520+0000","last_deep_scrub":"1057'78","last_deep_scrub_stamp":"2023-12-11T20:49:08.935520+0000","last_clean_scrub_stamp":"2023-12-11T20:49:08.935520+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:34:51.152246+0000","scrub_duration":0.11416854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1015,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:01.998619+0000","last_change":"2023-12-11T20:16:58.643847+0000","last_active":"2023-12-11T20:49:01.998619+0000","last_peered":"2023-12-11T20:49:01.998619+0000","last_clean":"2023-12-11T20:49:01.998619+0000","last_became_active":"2023-12-11T20:16:58.638544+0000","last_became_peered":"2023-12-11T20:16:58.638544+0000","last_unstale":"2023-12-11T20:49:01.998619+0000","last_undegraded":"2023-12-11T20:49:01.998619+0000","last_fullsized":"2023-12-11T20:49:01.998619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1005,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.451758+0000","last_change":"2023-12-11T20:18:12.933720+0000","last_active":"2023-12-11T20:49:02.451758+0000","last_peered":"2023-12-11T20:49:02.451758+0000","last_clean":"2023-12-11T20:49:02.451758+0000","last_became_active":"2023-12-11T20:18:12.924715+0000","last_became_peered":"2023-12-11T20:18:12.924715+0000","last_unstale":"2023-12-11T20:49:02.451758+0000","last_undegraded":"2023-12-11T20:49:02.451758+0000","last_fullsized":"2023-12-11T20:49:02.451758+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1015,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:09.206929+0000","last_change":"2023-12-11T20:16:58.036508+0000","last_active":"2023-12-11T20:49:09.206929+0000","last_peered":"2023-12-11T20:49:09.206929+0000","last_clean":"2023-12-11T20:49:09.206929+0000","last_became_active":"2023-12-11T20:16:58.008642+0000","last_became_peered":"2023-12-11T20:16:58.008642+0000","last_unstale":"2023-12-11T20:49:09.206929+0000","last_undegraded":"2023-12-11T20:49:09.206929+0000","last_fullsized":"2023-12-11T20:49:09.206929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1017,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.965629+0000","last_change":"2023-12-11T20:18:07.144229+0000","last_active":"2023-12-11T20:49:00.965629+0000","last_peered":"2023-12-11T20:49:00.965629+0000","last_clean":"2023-12-11T20:49:00.965629+0000","last_became_active":"2023-12-11T20:18:07.135656+0000","last_became_peered":"2023-12-11T20:18:07.135656+0000","last_unstale":"2023-12-11T20:49:00.965629+0000","last_undegraded":"2023-12-11T20:49:00.965629+0000","last_fullsized":"2023-12-11T20:49:00.965629+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:48:10.044525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1029,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.852656+0000","last_change":"2023-12-11T20:16:58.236956+0000","last_active":"2023-12-11T20:49:00.852656+0000","last_peered":"2023-12-11T20:49:00.852656+0000","last_clean":"2023-12-11T20:49:00.852656+0000","last_became_active":"2023-12-11T20:16:58.208987+0000","last_became_peered":"2023-12-11T20:16:58.208987+0000","last_unstale":"2023-12-11T20:49:00.852656+0000","last_undegraded":"2023-12-11T20:49:00.852656+0000","last_fullsized":"2023-12-11T20:49:00.852656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:57:37.650191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1024,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.988521+0000","last_change":"2023-12-11T20:18:01.362777+0000","last_active":"2023-12-11T20:49:00.988521+0000","last_peered":"2023-12-11T20:49:00.988521+0000","last_clean":"2023-12-11T20:49:00.988521+0000","last_became_active":"2023-12-11T20:18:01.354083+0000","last_became_peered":"2023-12-11T20:18:01.354083+0000","last_unstale":"2023-12-11T20:49:00.988521+0000","last_undegraded":"2023-12-11T20:49:00.988521+0000","last_fullsized":"2023-12-11T20:49:00.988521+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:40:18.542456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1015,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:01.991404+0000","last_change":"2023-12-11T20:16:58.356627+0000","last_active":"2023-12-11T20:49:01.991404+0000","last_peered":"2023-12-11T20:49:01.991404+0000","last_clean":"2023-12-11T20:49:01.991404+0000","last_became_active":"2023-12-11T20:16:58.349599+0000","last_became_peered":"2023-12-11T20:16:58.349599+0000","last_unstale":"2023-12-11T20:49:01.991404+0000","last_undegraded":"2023-12-11T20:49:01.991404+0000","last_fullsized":"2023-12-11T20:49:01.991404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1011,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.132566+0000","last_change":"2023-12-11T20:18:02.135407+0000","last_active":"2023-12-11T20:49:02.132566+0000","last_peered":"2023-12-11T20:49:02.132566+0000","last_clean":"2023-12-11T20:49:02.132566+0000","last_became_active":"2023-12-11T20:18:02.117789+0000","last_became_peered":"2023-12-11T20:18:02.117789+0000","last_unstale":"2023-12-11T20:49:02.132566+0000","last_undegraded":"2023-12-11T20:49:02.132566+0000","last_fullsized":"2023-12-11T20:49:02.132566+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.992399+0000","last_change":"2023-12-11T20:16:58.412827+0000","last_active":"2023-12-11T20:49:00.992399+0000","last_peered":"2023-12-11T20:49:00.992399+0000","last_clean":"2023-12-11T20:49:00.992399+0000","last_became_active":"2023-12-11T20:16:58.383357+0000","last_became_peered":"2023-12-11T20:16:58.383357+0000","last_unstale":"2023-12-11T20:49:00.992399+0000","last_undegraded":"2023-12-11T20:49:00.992399+0000","last_fullsized":"2023-12-11T20:49:00.992399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T20:59:10.927708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1016,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.998221+0000","last_change":"2023-12-11T20:18:13.718317+0000","last_active":"2023-12-11T20:49:00.998221+0000","last_peered":"2023-12-11T20:49:00.998221+0000","last_clean":"2023-12-11T20:49:00.998221+0000","last_became_active":"2023-12-11T20:18:13.700424+0000","last_became_peered":"2023-12-11T20:18:13.700424+0000","last_unstale":"2023-12-11T20:49:00.998221+0000","last_undegraded":"2023-12-11T20:49:00.998221+0000","last_fullsized":"2023-12-11T20:49:00.998221+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:57:09.221399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1015,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.002447+0000","last_change":"2023-12-11T20:16:58.680128+0000","last_active":"2023-12-11T20:49:02.002447+0000","last_peered":"2023-12-11T20:49:02.002447+0000","last_clean":"2023-12-11T20:49:02.002447+0000","last_became_active":"2023-12-11T20:16:58.674889+0000","last_became_peered":"2023-12-11T20:16:58.674889+0000","last_unstale":"2023-12-11T20:49:02.002447+0000","last_undegraded":"2023-12-11T20:49:02.002447+0000","last_fullsized":"2023-12-11T20:49:02.002447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1023,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.863245+0000","last_change":"2023-12-11T20:18:03.189633+0000","last_active":"2023-12-11T20:49:00.863245+0000","last_peered":"2023-12-11T20:49:00.863245+0000","last_clean":"2023-12-11T20:49:00.863245+0000","last_became_active":"2023-12-11T20:18:03.180691+0000","last_became_peered":"2023-12-11T20:18:03.180691+0000","last_unstale":"2023-12-11T20:49:00.863245+0000","last_undegraded":"2023-12-11T20:49:00.863245+0000","last_fullsized":"2023-12-11T20:49:00.863245+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:26:16.797482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1029,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:01.020907+0000","last_change":"2023-12-11T20:16:57.283249+0000","last_active":"2023-12-11T20:49:01.020907+0000","last_peered":"2023-12-11T20:49:01.020907+0000","last_clean":"2023-12-11T20:49:01.020907+0000","last_became_active":"2023-12-11T20:16:57.255904+0000","last_became_peered":"2023-12-11T20:16:57.255904+0000","last_unstale":"2023-12-11T20:49:01.020907+0000","last_undegraded":"2023-12-11T20:49:01.020907+0000","last_fullsized":"2023-12-11T20:49:01.020907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:17:34.118238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1016,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.990781+0000","last_change":"2023-12-11T20:18:33.771824+0000","last_active":"2023-12-11T20:49:00.990781+0000","last_peered":"2023-12-11T20:49:00.990781+0000","last_clean":"2023-12-11T20:49:00.990781+0000","last_became_active":"2023-12-11T20:18:33.762733+0000","last_became_peered":"2023-12-11T20:18:33.762733+0000","last_unstale":"2023-12-11T20:49:00.990781+0000","last_undegraded":"2023-12-11T20:49:00.990781+0000","last_fullsized":"2023-12-11T20:49:00.990781+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:53.954274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1014,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.012748+0000","last_change":"2023-12-11T20:16:58.076986+0000","last_active":"2023-12-11T20:49:02.012748+0000","last_peered":"2023-12-11T20:49:02.012748+0000","last_clean":"2023-12-11T20:49:02.012748+0000","last_became_active":"2023-12-11T20:16:58.071647+0000","last_became_peered":"2023-12-11T20:16:58.071647+0000","last_unstale":"2023-12-11T20:49:02.012748+0000","last_undegraded":"2023-12-11T20:49:02.012748+0000","last_fullsized":"2023-12-11T20:49:02.012748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:16:54.852869+0000","last_clean_scrub_stamp":"2023-12-11T20:16:54.852869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1008,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.301373+0000","last_change":"2023-12-11T20:18:11.891380+0000","last_active":"2023-12-11T20:49:02.301373+0000","last_peered":"2023-12-11T20:49:02.301373+0000","last_clean":"2023-12-11T20:49:02.301373+0000","last_became_active":"2023-12-11T20:18:11.885388+0000","last_became_peered":"2023-12-11T20:18:11.885388+0000","last_unstale":"2023-12-11T20:49:02.301373+0000","last_undegraded":"2023-12-11T20:49:02.301373+0000","last_fullsized":"2023-12-11T20:49:02.301373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1025,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:01.014475+0000","last_change":"2023-12-11T20:18:12.794888+0000","last_active":"2023-12-11T20:49:01.014475+0000","last_peered":"2023-12-11T20:49:01.014475+0000","last_clean":"2023-12-11T20:49:01.014475+0000","last_became_active":"2023-12-11T20:18:12.786107+0000","last_became_peered":"2023-12-11T20:18:12.786107+0000","last_unstale":"2023-12-11T20:49:01.014475+0000","last_undegraded":"2023-12-11T20:49:01.014475+0000","last_fullsized":"2023-12-11T20:49:01.014475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T23:00:25.310704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1025,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.866949+0000","last_change":"2023-12-11T20:18:00.474665+0000","last_active":"2023-12-11T20:49:00.866949+0000","last_peered":"2023-12-11T20:49:00.866949+0000","last_clean":"2023-12-11T20:49:00.866949+0000","last_became_active":"2023-12-11T20:18:00.469176+0000","last_became_peered":"2023-12-11T20:18:00.469176+0000","last_unstale":"2023-12-11T20:49:00.866949+0000","last_undegraded":"2023-12-11T20:49:00.866949+0000","last_fullsized":"2023-12-11T20:49:00.866949+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:24:10.853739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1011,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.886618+0000","last_change":"2023-12-11T20:18:05.144024+0000","last_active":"2023-12-11T20:49:00.886618+0000","last_peered":"2023-12-11T20:49:00.886618+0000","last_clean":"2023-12-11T20:49:00.886618+0000","last_became_active":"2023-12-11T20:18:05.138035+0000","last_became_peered":"2023-12-11T20:18:05.138035+0000","last_unstale":"2023-12-11T20:49:00.886618+0000","last_undegraded":"2023-12-11T20:49:00.886618+0000","last_fullsized":"2023-12-11T20:49:00.886618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:20:37.637313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1008,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.004145+0000","last_change":"2023-12-11T20:18:10.100025+0000","last_active":"2023-12-11T20:49:02.004145+0000","last_peered":"2023-12-11T20:49:02.004145+0000","last_clean":"2023-12-11T20:49:02.004145+0000","last_became_active":"2023-12-11T20:18:10.094694+0000","last_became_peered":"2023-12-11T20:18:10.094694+0000","last_unstale":"2023-12-11T20:49:02.004145+0000","last_undegraded":"2023-12-11T20:49:02.004145+0000","last_fullsized":"2023-12-11T20:49:02.004145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1015,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.984276+0000","last_change":"2023-12-11T20:18:07.073119+0000","last_active":"2023-12-11T20:49:00.984276+0000","last_peered":"2023-12-11T20:49:00.984276+0000","last_clean":"2023-12-11T20:49:00.984276+0000","last_became_active":"2023-12-11T20:18:07.062338+0000","last_became_peered":"2023-12-11T20:18:07.062338+0000","last_unstale":"2023-12-11T20:49:00.984276+0000","last_undegraded":"2023-12-11T20:49:00.984276+0000","last_fullsized":"2023-12-11T20:49:00.984276+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:48:41.166067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1009,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:01.993026+0000","last_change":"2023-12-11T20:18:19.678310+0000","last_active":"2023-12-11T20:49:01.993026+0000","last_peered":"2023-12-11T20:49:01.993026+0000","last_clean":"2023-12-11T20:49:01.993026+0000","last_became_active":"2023-12-11T20:18:19.665316+0000","last_became_peered":"2023-12-11T20:18:19.665316+0000","last_unstale":"2023-12-11T20:49:01.993026+0000","last_undegraded":"2023-12-11T20:49:01.993026+0000","last_fullsized":"2023-12-11T20:49:01.993026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1009,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:01.981318+0000","last_change":"2023-12-11T20:18:18.565871+0000","last_active":"2023-12-11T20:49:01.981318+0000","last_peered":"2023-12-11T20:49:01.981318+0000","last_clean":"2023-12-11T20:49:01.981318+0000","last_became_active":"2023-12-11T20:18:18.557249+0000","last_became_peered":"2023-12-11T20:18:18.557249+0000","last_unstale":"2023-12-11T20:49:01.981318+0000","last_undegraded":"2023-12-11T20:49:01.981318+0000","last_fullsized":"2023-12-11T20:49:01.981318+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1008,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:01.977529+0000","last_change":"2023-12-11T20:18:19.693007+0000","last_active":"2023-12-11T20:49:01.977529+0000","last_peered":"2023-12-11T20:49:01.977529+0000","last_clean":"2023-12-11T20:49:01.977529+0000","last_became_active":"2023-12-11T20:18:19.685187+0000","last_became_peered":"2023-12-11T20:18:19.685187+0000","last_unstale":"2023-12-11T20:49:01.977529+0000","last_undegraded":"2023-12-11T20:49:01.977529+0000","last_fullsized":"2023-12-11T20:49:01.977529+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1010,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.136442+0000","last_change":"2023-12-11T20:18:00.559982+0000","last_active":"2023-12-11T20:49:02.136442+0000","last_peered":"2023-12-11T20:49:02.136442+0000","last_clean":"2023-12-11T20:49:02.136442+0000","last_became_active":"2023-12-11T20:18:00.554682+0000","last_became_peered":"2023-12-11T20:18:00.554682+0000","last_unstale":"2023-12-11T20:49:02.136442+0000","last_undegraded":"2023-12-11T20:49:02.136442+0000","last_fullsized":"2023-12-11T20:49:02.136442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1015,"reported_epoch":1056,"state":"active+clean","last_fresh":"2023-12-11T20:49:00.996601+0000","last_change":"2023-12-11T20:18:13.715085+0000","last_active":"2023-12-11T20:49:00.996601+0000","last_peered":"2023-12-11T20:49:00.996601+0000","last_clean":"2023-12-11T20:49:00.996601+0000","last_became_active":"2023-12-11T20:18:13.689623+0000","last_became_peered":"2023-12-11T20:18:13.689623+0000","last_unstale":"2023-12-11T20:49:00.996601+0000","last_undegraded":"2023-12-11T20:49:00.996601+0000","last_fullsized":"2023-12-11T20:49:00.996601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:14:13.791518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1008,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.000867+0000","last_change":"2023-12-11T20:18:10.128111+0000","last_active":"2023-12-11T20:49:02.000867+0000","last_peered":"2023-12-11T20:49:02.000867+0000","last_clean":"2023-12-11T20:49:02.000867+0000","last_became_active":"2023-12-11T20:18:10.112911+0000","last_became_peered":"2023-12-11T20:18:10.112911+0000","last_unstale":"2023-12-11T20:49:02.000867+0000","last_undegraded":"2023-12-11T20:49:02.000867+0000","last_fullsized":"2023-12-11T20:49:02.000867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1003,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.456573+0000","last_change":"2023-12-11T20:18:13.209324+0000","last_active":"2023-12-11T20:49:02.456573+0000","last_peered":"2023-12-11T20:49:02.456573+0000","last_clean":"2023-12-11T20:49:02.456573+0000","last_became_active":"2023-12-11T20:18:13.203517+0000","last_became_peered":"2023-12-11T20:18:13.203517+0000","last_unstale":"2023-12-11T20:49:02.456573+0000","last_undegraded":"2023-12-11T20:49:02.456573+0000","last_fullsized":"2023-12-11T20:49:02.456573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719477125,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69332,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302508,"statfs":{"total":96636764160,"available":96565768192,"internally_reserved":0,"allocated":2105344,"data_stored":556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":17.427,"5min":10.535,"15min":14.893000000000001},"min":{"1min":4.5739999999999998,"5min":4.5739999999999998,"15min":4.2510000000000003},"max":{"1min":112.848,"5min":112.848,"15min":221.21299999999999},"last":7.2779999999999996},{"interface":"front","average":{"1min":17.669,"5min":11.170999999999999,"15min":14.507999999999999},"min":{"1min":6.2409999999999997,"5min":5.3739999999999997,"15min":4.7930000000000001},"max":{"1min":94.400999999999996,"5min":94.400999999999996,"15min":164.01900000000001},"last":13.382999999999999}]},{"osd":1,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":21.391999999999999,"5min":12.878,"15min":16.920999999999999},"min":{"1min":7.6660000000000004,"5min":6.8419999999999996,"15min":5.0380000000000003},"max":{"1min":117.63,"5min":117.63,"15min":195.87200000000001},"last":12.720000000000001},{"interface":"front","average":{"1min":26.209,"5min":17.108000000000001,"15min":20.576000000000001},"min":{"1min":10.843999999999999,"5min":9.6649999999999991,"15min":6.0300000000000002},"max":{"1min":118.309,"5min":118.309,"15min":190.53899999999999},"last":14.65}]},{"osd":2,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":15.789999999999999,"5min":11.457000000000001,"15min":18.841999999999999},"min":{"1min":6.4089999999999998,"5min":6.4089999999999998,"15min":5.8259999999999996},"max":{"1min":67.793000000000006,"5min":67.793000000000006,"15min":282.79000000000002},"last":8.2129999999999992},{"interface":"front","average":{"1min":21.603000000000002,"5min":15.961,"15min":21.614000000000001},"min":{"1min":8.1829999999999998,"5min":8.1829999999999998,"15min":8.0570000000000004},"max":{"1min":68.638999999999996,"5min":68.638999999999996,"15min":200.148},"last":12.06}]},{"osd":3,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.715,"5min":15.484999999999999,"15min":19.199999999999999},"min":{"1min":9.6660000000000004,"5min":6.5270000000000001,"15min":5.3029999999999999},"max":{"1min":113.89400000000001,"5min":113.89400000000001,"15min":206.11799999999999},"last":18.210000000000001},{"interface":"front","average":{"1min":22.518000000000001,"5min":14.561,"15min":20.105},"min":{"1min":6.7809999999999997,"5min":6.7809999999999997,"15min":6.7809999999999997},"max":{"1min":140.172,"5min":140.172,"15min":171.63},"last":5.976}]},{"osd":4,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":24.693000000000001,"5min":16.344999999999999,"15min":18.094000000000001},"min":{"1min":8.5180000000000007,"5min":7.6289999999999996,"15min":6.577},"max":{"1min":110.98399999999999,"5min":110.98399999999999,"15min":188.26599999999999},"last":11.372},{"interface":"front","average":{"1min":25.422999999999998,"5min":17.34,"15min":21.085000000000001},"min":{"1min":7.9279999999999999,"5min":7.8470000000000004,"15min":7.5300000000000002},"max":{"1min":144.084,"5min":144.084,"15min":172.57400000000001},"last":14.028}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":19.388999999999999,"5min":14.335000000000001,"15min":18.338999999999999},"min":{"1min":8.4640000000000004,"5min":7.4800000000000004,"15min":6.141},"max":{"1min":62.637,"5min":62.637,"15min":201.25399999999999},"last":16.062000000000001},{"interface":"front","average":{"1min":24.920999999999999,"5min":17.582999999999998,"15min":21.760000000000002},"min":{"1min":10.914999999999999,"5min":8.9529999999999994,"15min":8.423},"max":{"1min":92.804000000000002,"5min":92.804000000000002,"15min":200.60599999999999},"last":9.7759999999999998}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.809999999999999,"5min":18.367999999999999,"15min":21.443000000000001},"min":{"1min":10.18,"5min":8.4909999999999997,"15min":7.899},"max":{"1min":76.634,"5min":76.634,"15min":207.72200000000001},"last":10.368},{"interface":"front","average":{"1min":22.077000000000002,"5min":17.238,"15min":22.446999999999999},"min":{"1min":9.6129999999999995,"5min":9.3019999999999996,"15min":9.3019999999999996},"max":{"1min":69.144000000000005,"5min":69.144000000000005,"15min":193.43899999999999},"last":15.34}]}]},{"osd":3,"up_from":16,"seq":68719477125,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65168,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306672,"statfs":{"total":96636764160,"available":96570032128,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":21.152000000000001,"5min":11.818,"15min":15.324999999999999},"min":{"1min":6.3159999999999998,"5min":4.4720000000000004,"15min":4.1600000000000001},"max":{"1min":102.907,"5min":102.907,"15min":266.75400000000002},"last":4.9790000000000001},{"interface":"front","average":{"1min":22.907,"5min":12.157999999999999,"15min":16.690999999999999},"min":{"1min":5.4100000000000001,"5min":5.4100000000000001,"15min":4.5330000000000004},"max":{"1min":110.068,"5min":110.068,"15min":228.12899999999999},"last":10.189}]},{"osd":1,"last update":"Mon Dec 11 20:48:39 2023","interfaces":[{"interface":"back","average":{"1min":42.024000000000001,"5min":17.748999999999999,"15min":16.431000000000001},"min":{"1min":7.1529999999999996,"5min":6.7489999999999997,"15min":5.6180000000000003},"max":{"1min":170.31200000000001,"5min":170.31200000000001,"15min":179.11000000000001},"last":11.949999999999999},{"interface":"front","average":{"1min":50.826000000000001,"5min":19.158999999999999,"15min":18.561},"min":{"1min":8.9849999999999994,"5min":7.0529999999999999,"15min":6.508},"max":{"1min":215.578,"5min":215.578,"15min":215.578},"last":11.204000000000001}]},{"osd":2,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":27.824000000000002,"5min":15.458,"15min":18.983000000000001},"min":{"1min":8.6820000000000004,"5min":8.1479999999999997,"15min":7.1150000000000002},"max":{"1min":146.12700000000001,"5min":146.12700000000001,"15min":226.56299999999999},"last":9.9060000000000006},{"interface":"front","average":{"1min":30.817,"5min":17.829000000000001,"15min":21.931000000000001},"min":{"1min":11.138999999999999,"5min":10.387,"15min":9.5990000000000002},"max":{"1min":156.46199999999999,"5min":156.46199999999999,"15min":227.40899999999999},"last":14.147}]},{"osd":4,"last update":"Mon Dec 11 20:48:12 2023","interfaces":[{"interface":"back","average":{"1min":26.216999999999999,"5min":17.390000000000001,"15min":20.870000000000001},"min":{"1min":10.401999999999999,"5min":10.08,"15min":9.6539999999999999},"max":{"1min":91.260000000000005,"5min":91.260000000000005,"15min":246.178},"last":15.842000000000001},{"interface":"front","average":{"1min":21.318999999999999,"5min":13.746,"15min":16.047999999999998},"min":{"1min":7.742,"5min":7.0510000000000002,"15min":7.0510000000000002},"max":{"1min":82.622,"5min":82.622,"15min":173.66800000000001},"last":12.776}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":40.828000000000003,"5min":19.824000000000002,"15min":23.663},"min":{"1min":10.592000000000001,"5min":7.5250000000000004,"15min":7.5250000000000004},"max":{"1min":119.569,"5min":119.569,"15min":246.55000000000001},"last":15.134},{"interface":"front","average":{"1min":49.973999999999997,"5min":16.795999999999999,"15min":17.120999999999999},"min":{"1min":6.2610000000000001,"5min":5.4429999999999996,"15min":5.4429999999999996},"max":{"1min":203.21799999999999,"5min":203.21799999999999,"15min":205.15799999999999},"last":7.3520000000000003}]},{"osd":6,"last update":"Mon Dec 11 20:48:12 2023","interfaces":[{"interface":"back","average":{"1min":30.149999999999999,"5min":19.039000000000001,"15min":25.846},"min":{"1min":11.76,"5min":10.831,"15min":9.9320000000000004},"max":{"1min":114.998,"5min":114.998,"15min":266.13499999999999},"last":13.506},{"interface":"front","average":{"1min":28.504000000000001,"5min":17.774999999999999,"15min":23.965},"min":{"1min":11.42,"5min":9.8339999999999996,"15min":8.391},"max":{"1min":115.377,"5min":115.377,"15min":208.00299999999999},"last":17.268000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":60.445,"5min":26.123000000000001,"15min":21.013999999999999},"min":{"1min":12.372,"5min":12.295999999999999,"15min":6.4690000000000003},"max":{"1min":260.45999999999998,"5min":260.45999999999998,"15min":260.45999999999998},"last":18.196000000000002},{"interface":"front","average":{"1min":44.287999999999997,"5min":22.015999999999998,"15min":25.881},"min":{"1min":12.273999999999999,"5min":10.863,"15min":10.863},"max":{"1min":172.559,"5min":172.559,"15min":264.99299999999999},"last":19.277999999999999}]}]},{"osd":2,"up_from":16,"seq":68719477124,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65168,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306672,"statfs":{"total":96636764160,"available":96570032128,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":20.559000000000001,"5min":12.036,"15min":15.512},"min":{"1min":5.0780000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"max":{"1min":89.881,"5min":89.881,"15min":200.447},"last":12.77},{"interface":"front","average":{"1min":19.225000000000001,"5min":11.49,"15min":16.253},"min":{"1min":5.6619999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":92.114000000000004,"5min":92.114000000000004,"15min":198.22900000000001},"last":8.0709999999999997}]},{"osd":1,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":21.713000000000001,"5min":13.699,"15min":16.245999999999999},"min":{"1min":7.0590000000000002,"5min":5.7869999999999999,"15min":5.7869999999999999},"max":{"1min":89.183999999999997,"5min":89.183999999999997,"15min":143.56},"last":13.542999999999999},{"interface":"front","average":{"1min":27.739999999999998,"5min":15.618,"15min":17.497},"min":{"1min":6.1879999999999997,"5min":6.1879999999999997,"15min":6.1879999999999997},"max":{"1min":175.16200000000001,"5min":175.16200000000001,"15min":175.16200000000001},"last":9.3810000000000002}]},{"osd":3,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":19.030999999999999,"5min":11.109999999999999,"15min":13.959},"min":{"1min":6.0780000000000003,"5min":5.7960000000000003,"15min":5.3550000000000004},"max":{"1min":99.146000000000001,"5min":99.146000000000001,"15min":171.71700000000001},"last":16.969000000000001},{"interface":"front","average":{"1min":21.315999999999999,"5min":15.250999999999999,"15min":17.931000000000001},"min":{"1min":8.6479999999999997,"5min":8.0220000000000002,"15min":5.9749999999999996},"max":{"1min":82.924999999999997,"5min":82.924999999999997,"15min":182.63399999999999},"last":8.6479999999999997}]},{"osd":4,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":32.999000000000002,"5min":17.382999999999999,"15min":19.341999999999999},"min":{"1min":6.694,"5min":6.694,"15min":6.694},"max":{"1min":192.851,"5min":192.851,"15min":192.851},"last":6.694},{"interface":"front","average":{"1min":40.692999999999998,"5min":21.776,"15min":24.178999999999998},"min":{"1min":11.397,"5min":10.923999999999999,"15min":10.285},"max":{"1min":259.05900000000003,"5min":259.05900000000003,"15min":259.05900000000003},"last":16.231000000000002}]},{"osd":5,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":49.709000000000003,"5min":22.446999999999999,"15min":21.960999999999999},"min":{"1min":7.8159999999999998,"5min":7.8159999999999998,"15min":7.8159999999999998},"max":{"1min":364.11200000000002,"5min":364.11200000000002,"15min":364.11200000000002},"last":10.672000000000001},{"interface":"front","average":{"1min":39.561999999999998,"5min":20.593,"15min":24.591999999999999},"min":{"1min":10.625999999999999,"5min":9.1349999999999998,"15min":8.5020000000000007},"max":{"1min":296.74000000000001,"5min":296.74000000000001,"15min":296.74000000000001},"last":12.131}]},{"osd":6,"last update":"Mon Dec 11 20:48:18 2023","interfaces":[{"interface":"back","average":{"1min":24.114999999999998,"5min":14.093999999999999,"15min":17.513999999999999},"min":{"1min":6.6799999999999997,"5min":6.6799999999999997,"15min":6.0990000000000002},"max":{"1min":251.19300000000001,"5min":251.19300000000001,"15min":251.19300000000001},"last":9.9849999999999994},{"interface":"front","average":{"1min":17.216000000000001,"5min":16.463999999999999,"15min":21.641999999999999},"min":{"1min":12.948,"5min":9.4960000000000004,"15min":9.4469999999999992},"max":{"1min":30.542000000000002,"5min":30.542000000000002,"15min":181.96299999999999},"last":14.167}]},{"osd":7,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":38.201999999999998,"5min":22.239999999999998,"15min":24.962},"min":{"1min":13.659000000000001,"5min":10.379,"15min":10.379},"max":{"1min":156.934,"5min":156.934,"15min":197.27199999999999},"last":15.59},{"interface":"front","average":{"1min":31.547999999999998,"5min":17.236000000000001,"15min":19.617000000000001},"min":{"1min":9.9179999999999993,"5min":8.0199999999999996,"15min":7.085},"max":{"1min":173.654,"5min":173.654,"15min":233.57400000000001},"last":18.109000000000002}]}]},{"osd":1,"up_from":16,"seq":68719477125,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69728,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302112,"statfs":{"total":96636764160,"available":96565362688,"internally_reserved":0,"allocated":2502656,"data_stored":954021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":7.8470000000000004,"15min":13.646000000000001},"min":{"1min":4.6870000000000003,"5min":4.0949999999999998,"15min":3.903},"max":{"1min":72.888000000000005,"5min":72.888000000000005,"15min":283.91899999999998},"last":6.1349999999999998},{"interface":"front","average":{"1min":17.266999999999999,"5min":9.9909999999999997,"15min":14.855},"min":{"1min":5.8250000000000002,"5min":4.9340000000000002,"15min":4.6879999999999997},"max":{"1min":99.680000000000007,"5min":99.680000000000007,"15min":391.19999999999999},"last":12.395}]},{"osd":2,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":18.952999999999999,"5min":13.276,"15min":19.867999999999999},"min":{"1min":7.7510000000000003,"5min":6.0300000000000002,"15min":6.0300000000000002},"max":{"1min":84.167000000000002,"5min":84.167000000000002,"15min":389.66800000000001},"last":14.132},{"interface":"front","average":{"1min":19.573,"5min":14.933999999999999,"15min":20.850000000000001},"min":{"1min":9.2989999999999995,"5min":9.2590000000000003,"15min":6.7670000000000003},"max":{"1min":100.836,"5min":100.836,"15min":376.82100000000003},"last":15.052}]},{"osd":3,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":20.777999999999999,"5min":12.682,"15min":24.100999999999999},"min":{"1min":6.9080000000000004,"5min":5.7610000000000001,"15min":5.7610000000000001},"max":{"1min":101.74299999999999,"5min":101.74299999999999,"15min":421.89400000000001},"last":11.323},{"interface":"front","average":{"1min":29.260000000000002,"5min":16.670000000000002,"15min":20.762},"min":{"1min":10.404,"5min":5.7220000000000004,"15min":5.7220000000000004},"max":{"1min":175.047,"5min":175.047,"15min":293.06099999999998},"last":15.840999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":12.462999999999999,"15min":18.878},"min":{"1min":6.6740000000000004,"5min":6.1630000000000003,"15min":5.952},"max":{"1min":120.863,"5min":120.863,"15min":342.28199999999998},"last":8.1869999999999994},{"interface":"front","average":{"1min":33.405000000000001,"5min":18.756,"15min":23.010000000000002},"min":{"1min":12.538,"5min":10.499000000000001,"15min":6.7039999999999997},"max":{"1min":187.15100000000001,"5min":187.15100000000001,"15min":289.68799999999999},"last":13.529}]},{"osd":5,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":25.463999999999999,"5min":17.777000000000001,"15min":22.192},"min":{"1min":11.183,"5min":11.183,"15min":7.4859999999999998},"max":{"1min":183.84700000000001,"5min":183.84700000000001,"15min":556.32500000000005},"last":11.832000000000001},{"interface":"front","average":{"1min":27.771000000000001,"5min":19.635999999999999,"15min":24.861000000000001},"min":{"1min":12.422000000000001,"5min":12.422000000000001,"15min":9.5739999999999998},"max":{"1min":186.06700000000001,"5min":186.06700000000001,"15min":309.16399999999999},"last":18.981000000000002}]},{"osd":6,"last update":"Mon Dec 11 20:48:43 2023","interfaces":[{"interface":"back","average":{"1min":23.209,"5min":13.757,"15min":20.975000000000001},"min":{"1min":8.6910000000000007,"5min":7.5270000000000001,"15min":6.7590000000000003},"max":{"1min":169.429,"5min":169.429,"15min":372.90899999999999},"last":9.5589999999999993},{"interface":"front","average":{"1min":29.114999999999998,"5min":19.785,"15min":27.733000000000001},"min":{"1min":12.670999999999999,"5min":12.670999999999999,"15min":11.641999999999999},"max":{"1min":197.87299999999999,"5min":197.87299999999999,"15min":297.71499999999997},"last":19.998000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":20.015000000000001,"5min":13.327,"15min":22.792999999999999},"min":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"max":{"1min":109.782,"5min":109.782,"15min":426.10700000000003},"last":7.5289999999999999},{"interface":"front","average":{"1min":33.633000000000003,"5min":21.547000000000001,"15min":29.686},"min":{"1min":13.068,"5min":13.068,"15min":11.789},"max":{"1min":189.21199999999999,"5min":189.21199999999999,"15min":331.24799999999999},"last":20.805}]}]},{"osd":0,"up_from":16,"seq":68719477127,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":11.036,"15min":16.446000000000002},"min":{"1min":4.609,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":98.932000000000002,"5min":98.932000000000002,"15min":192.989},"last":29.594000000000001},{"interface":"front","average":{"1min":20.565000000000001,"5min":11.683999999999999,"15min":17.998999999999999},"min":{"1min":6.1719999999999997,"5min":5.2220000000000004,"15min":4.7300000000000004},"max":{"1min":101.66,"5min":101.66,"15min":288.91800000000001},"last":54.594000000000001}]},{"osd":2,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":22.863,"5min":13.515000000000001,"15min":17.582999999999998},"min":{"1min":7.694,"5min":7.0620000000000003,"15min":5.2190000000000003},"max":{"1min":96.489999999999995,"5min":96.489999999999995,"15min":294.733},"last":14.007},{"interface":"front","average":{"1min":29.260999999999999,"5min":17.803999999999998,"15min":23.199999999999999},"min":{"1min":11.227,"5min":9.9109999999999996,"15min":7.8630000000000004},"max":{"1min":126.831,"5min":126.831,"15min":306.08100000000002},"last":18.908000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.068000000000001,"5min":13.037000000000001,"15min":15.56},"min":{"1min":6.1769999999999996,"5min":5.9829999999999997,"15min":5.5979999999999999},"max":{"1min":160.999,"5min":160.999,"15min":166.49100000000001},"last":15.949},{"interface":"front","average":{"1min":29.303000000000001,"5min":17.632999999999999,"15min":20.312999999999999},"min":{"1min":10.973000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"max":{"1min":177.35400000000001,"5min":177.35400000000001,"15min":177.35400000000001},"last":11.063000000000001}]},{"osd":4,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.286000000000001,"5min":16.172999999999998,"15min":19.902999999999999},"min":{"1min":6.5970000000000004,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":169.92599999999999,"5min":169.92599999999999,"15min":169.92599999999999},"last":9.1950000000000003},{"interface":"front","average":{"1min":31.260999999999999,"5min":19.547000000000001,"15min":20.800000000000001},"min":{"1min":10.371,"5min":10.371,"15min":6.6859999999999999},"max":{"1min":149.65600000000001,"5min":149.65600000000001,"15min":153.42500000000001},"last":13.417999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.992999999999999,"5min":13.356,"15min":15.753},"min":{"1min":7.0940000000000003,"5min":6.5999999999999996,"15min":6.234},"max":{"1min":110.476,"5min":110.476,"15min":210.374},"last":9.6920000000000002},{"interface":"front","average":{"1min":30.498999999999999,"5min":16.635999999999999,"15min":20.227},"min":{"1min":8.7940000000000005,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":110.836,"5min":110.836,"15min":267.03199999999998},"last":11.890000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":24.010999999999999,"5min":14.885,"15min":18.350999999999999},"min":{"1min":9.0850000000000009,"5min":8.0050000000000008,"15min":6.4790000000000001},"max":{"1min":111.423,"5min":111.423,"15min":161.09899999999999},"last":14.752000000000001},{"interface":"front","average":{"1min":33.442999999999998,"5min":19.559000000000001,"15min":23.120000000000001},"min":{"1min":11.138999999999999,"5min":10.305999999999999,"15min":9.4440000000000008},"max":{"1min":186.089,"5min":186.089,"15min":246.738},"last":17.762}]},{"osd":7,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.001000000000001,"5min":19.183,"15min":21.738},"min":{"1min":9.4049999999999994,"5min":9.4049999999999994,"15min":8.8789999999999996},"max":{"1min":187.00299999999999,"5min":187.00299999999999,"15min":187.00299999999999},"last":20.001999999999999},{"interface":"front","average":{"1min":29.199999999999999,"5min":19.706,"15min":23.911999999999999},"min":{"1min":12.224,"5min":11.782999999999999,"15min":7.7640000000000002},"max":{"1min":127.563,"5min":127.563,"15min":264.45100000000002},"last":16.904}]}]},{"osd":4,"up_from":14,"seq":60129542530,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65172,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306668,"statfs":{"total":96636764160,"available":96570028032,"internally_reserved":0,"allocated":2105344,"data_stored":556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":13.959,"5min":10.147,"15min":15.048999999999999},"min":{"1min":4.5049999999999999,"5min":4.4409999999999998,"15min":4.1879999999999997},"max":{"1min":40.877000000000002,"5min":40.877000000000002,"15min":248.19900000000001},"last":6.7460000000000004},{"interface":"front","average":{"1min":17.295999999999999,"5min":10.696999999999999,"15min":13.792},"min":{"1min":5.1589999999999998,"5min":5.1059999999999999,"15min":4.1449999999999996},"max":{"1min":77.028000000000006,"5min":77.028000000000006,"15min":169.541},"last":6.1189999999999998}]},{"osd":1,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":23.628,"5min":14.050000000000001,"15min":18.832000000000001},"min":{"1min":7.1269999999999998,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":116.298,"5min":116.298,"15min":310.53500000000003},"last":12.593999999999999},{"interface":"front","average":{"1min":21.18,"5min":13.920999999999999,"15min":18.367999999999999},"min":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":4.7380000000000004},"max":{"1min":109.753,"5min":109.753,"15min":269.65100000000001},"last":11.279999999999999}]},{"osd":2,"last update":"Mon Dec 11 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":14.571999999999999,"5min":13.747999999999999,"15min":18.155000000000001},"min":{"1min":8.3030000000000008,"5min":8.3030000000000008,"15min":5.0830000000000002},"max":{"1min":24.594000000000001,"5min":24.594000000000001,"15min":255.31700000000001},"last":14.013},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.42,"15min":17.867000000000001},"min":{"1min":6.6429999999999998,"5min":6.6310000000000002,"15min":5.4199999999999999},"max":{"1min":16.919,"5min":25.324999999999999,"15min":291.67899999999997},"last":9.9309999999999992}]},{"osd":3,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":13.087,"5min":12.433999999999999,"15min":18.640999999999998},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.2460000000000004},"max":{"1min":23.73,"5min":23.73,"15min":260.55099999999999},"last":11.962999999999999},{"interface":"front","average":{"1min":16.361000000000001,"5min":15.35,"15min":22.067},"min":{"1min":9.2949999999999999,"5min":8.1969999999999992,"15min":7.0030000000000001},"max":{"1min":20.364999999999998,"5min":21.821000000000002,"15min":320.53899999999999},"last":17.827000000000002}]},{"osd":5,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":26.222000000000001,"5min":18.422999999999998,"15min":21.838999999999999},"min":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":7.7359999999999998},"max":{"1min":132.37799999999999,"5min":132.37799999999999,"15min":298.69499999999999},"last":18.623999999999999},{"interface":"front","average":{"1min":22.783999999999999,"5min":15.782,"15min":21.024999999999999},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":81.689999999999998,"5min":81.689999999999998,"15min":309.88200000000001},"last":13.282999999999999}]},{"osd":6,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":16.067,"5min":14.606999999999999,"15min":18.742000000000001},"min":{"1min":10.874000000000001,"5min":7.6509999999999998,"15min":5.4459999999999997},"max":{"1min":30.559000000000001,"5min":30.559000000000001,"15min":290.57600000000002},"last":14.791},{"interface":"front","average":{"1min":12.907999999999999,"5min":14.144,"15min":18.532},"min":{"1min":7.3540000000000001,"5min":7.3540000000000001,"15min":6.5389999999999997},"max":{"1min":27.225999999999999,"5min":27.225999999999999,"15min":285.52499999999998},"last":9.1080000000000005}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":21.823,"5min":17.431000000000001,"15min":21.483000000000001},"min":{"1min":10.587,"5min":10.176,"15min":7.2599999999999998},"max":{"1min":69.731999999999999,"5min":69.731999999999999,"15min":297.63299999999998},"last":16.864999999999998},{"interface":"front","average":{"1min":23.029,"5min":16.969999999999999,"15min":23.030000000000001},"min":{"1min":10.496,"5min":7.3639999999999999,"15min":7.3639999999999999},"max":{"1min":98.728999999999999,"5min":98.728999999999999,"15min":364.18799999999999},"last":10.599}]}]},{"osd":7,"up_from":13,"seq":55834575238,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65172,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306668,"statfs":{"total":96636764160,"available":96570028032,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:16 2023","interfaces":[{"interface":"back","average":{"1min":9.4580000000000002,"5min":10.074999999999999,"15min":17.242999999999999},"min":{"1min":4.3600000000000003,"5min":4.3600000000000003,"15min":3.8969999999999998},"max":{"1min":15.532999999999999,"5min":16.571000000000002,"15min":280.62200000000001},"last":6.5510000000000002},{"interface":"front","average":{"1min":9.0850000000000009,"5min":9.4939999999999998,"15min":17.504000000000001},"min":{"1min":6.3550000000000004,"5min":5.5990000000000002,"15min":4.9800000000000004},"max":{"1min":12.73,"5min":24.454999999999998,"15min":294.44299999999998},"last":13.398}]},{"osd":1,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":11.183999999999999,"5min":11.531000000000001,"15min":16.939},"min":{"1min":6.4340000000000002,"5min":5.5519999999999996,"15min":5.4489999999999998},"max":{"1min":18.149999999999999,"5min":18.594000000000001,"15min":217.40100000000001},"last":14.975},{"interface":"front","average":{"1min":12.154,"5min":12.77,"15min":18.738},"min":{"1min":8.1039999999999992,"5min":6.3730000000000002,"15min":6.3730000000000002},"max":{"1min":16.456,"5min":18.507999999999999,"15min":253.15799999999999},"last":9.9960000000000004}]},{"osd":2,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":12.81,"5min":13.550000000000001,"15min":18.916},"min":{"1min":7.8259999999999996,"5min":7.7839999999999998,"15min":6.319},"max":{"1min":16.373000000000001,"5min":20.346,"15min":275.31},"last":11.265000000000001},{"interface":"front","average":{"1min":15.932,"5min":16.280000000000001,"15min":24.547999999999998},"min":{"1min":11.714,"5min":11.714,"15min":11.016},"max":{"1min":19.315999999999999,"5min":22.353000000000002,"15min":399.96100000000001},"last":15.879}]},{"osd":3,"last update":"Mon Dec 11 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":8.4879999999999995,"5min":8.3040000000000003,"15min":13.114000000000001},"min":{"1min":5.1879999999999997,"5min":5.1879999999999997,"15min":5.1879999999999997},"max":{"1min":13.698,"5min":16.690999999999999,"15min":239.00200000000001},"last":8.9440000000000008},{"interface":"front","average":{"1min":14.753,"5min":15.654999999999999,"15min":21.408000000000001},"min":{"1min":10.894,"5min":9.6389999999999993,"15min":9.2240000000000002},"max":{"1min":17.164000000000001,"5min":21.308,"15min":263.565},"last":14.31}]},{"osd":4,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":27.109999999999999,"5min":15.597,"15min":16.378},"min":{"1min":9.6489999999999991,"5min":6.516,"15min":6.1900000000000004},"max":{"1min":166.67099999999999,"5min":166.67099999999999,"15min":201.459},"last":17.446000000000002},{"interface":"front","average":{"1min":22.686,"5min":13.342000000000001,"15min":17.608000000000001},"min":{"1min":7.4580000000000002,"5min":6.2510000000000003,"15min":6.2510000000000003},"max":{"1min":124.88,"5min":124.88,"15min":203.36600000000001},"last":7.907}]},{"osd":5,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":39.771000000000001,"5min":21.721,"15min":26.289000000000001},"min":{"1min":12.252000000000001,"5min":10.476000000000001,"15min":9.7810000000000006},"max":{"1min":206.904,"5min":206.904,"15min":268.98000000000002},"last":11.843999999999999},{"interface":"front","average":{"1min":28.629000000000001,"5min":16.504999999999999,"15min":19.289999999999999},"min":{"1min":8.9410000000000007,"5min":8.2639999999999993,"15min":7.6360000000000001},"max":{"1min":127.8,"5min":127.8,"15min":283.61900000000003},"last":16.713999999999999}]},{"osd":6,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":22.581,"5min":16.634,"15min":21.099},"min":{"1min":12.592000000000001,"5min":8.8339999999999996,"15min":6.6479999999999997},"max":{"1min":115.38800000000001,"5min":115.38800000000001,"15min":282.71199999999999},"last":18.283999999999999},{"interface":"front","average":{"1min":25.696999999999999,"5min":19.786000000000001,"15min":25.102},"min":{"1min":15.474,"5min":10.999000000000001,"15min":9.7569999999999997},"max":{"1min":126.97199999999999,"5min":126.97199999999999,"15min":260.13400000000001},"last":12.372999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575236,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69716,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302124,"statfs":{"total":96636764160,"available":96565374976,"internally_reserved":0,"allocated":2502656,"data_stored":954021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":13.065,"5min":10.132999999999999,"15min":13.657999999999999},"min":{"1min":4.4960000000000004,"5min":4.4509999999999996,"15min":4.3849999999999998},"max":{"1min":52.319000000000003,"5min":52.319000000000003,"15min":121.80500000000001},"last":11.082000000000001},{"interface":"front","average":{"1min":13.058999999999999,"5min":9.7390000000000008,"15min":16.152999999999999},"min":{"1min":6.3780000000000001,"5min":4.5369999999999999,"15min":4.4349999999999996},"max":{"1min":49.723999999999997,"5min":49.723999999999997,"15min":307.66000000000003},"last":9.6129999999999995}]},{"osd":1,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":12.073,"5min":9.8230000000000004,"15min":13.728},"min":{"1min":6.1070000000000002,"5min":5.3449999999999998,"15min":4.5490000000000004},"max":{"1min":57.738999999999997,"5min":57.738999999999997,"15min":131.70500000000001},"last":7.7130000000000001},{"interface":"front","average":{"1min":14.811,"5min":11.605,"15min":16.902000000000001},"min":{"1min":7.4320000000000004,"5min":6.0609999999999999,"15min":5.7169999999999996},"max":{"1min":60.664999999999999,"5min":60.664999999999999,"15min":202.07300000000001},"last":12.958}]},{"osd":2,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.559000000000001,"5min":15.26,"15min":18.596},"min":{"1min":8.6720000000000006,"5min":7.7460000000000004,"15min":6.9470000000000001},"max":{"1min":109.023,"5min":109.023,"15min":203.67699999999999},"last":7.3609999999999998},{"interface":"front","average":{"1min":23.201000000000001,"5min":15.288,"15min":19.117000000000001},"min":{"1min":9.0129999999999999,"5min":7.8760000000000003,"15min":6.8719999999999999},"max":{"1min":93.819999999999993,"5min":93.819999999999993,"15min":211.67500000000001},"last":13.554}]},{"osd":3,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":15.050000000000001,"15min":19.510000000000002},"min":{"1min":8.3170000000000002,"5min":8.0700000000000003,"15min":7.548},"max":{"1min":58.795000000000002,"5min":58.795000000000002,"15min":167.095},"last":8.4299999999999997},{"interface":"front","average":{"1min":19.649000000000001,"5min":15.371,"15min":19.989000000000001},"min":{"1min":10.052,"5min":8.1690000000000005,"15min":7.6769999999999996},"max":{"1min":68.209999999999994,"5min":68.209999999999994,"15min":207.58799999999999},"last":14.151}]},{"osd":4,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.547000000000001,"5min":16.661000000000001,"15min":19.132000000000001},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":6.8360000000000003},"max":{"1min":69.195999999999998,"5min":69.195999999999998,"15min":132.64500000000001},"last":11.731999999999999},{"interface":"front","average":{"1min":33.280999999999999,"5min":20.861000000000001,"15min":23.035},"min":{"1min":14.875999999999999,"5min":9.9390000000000001,"15min":8.2590000000000003},"max":{"1min":199.828,"5min":199.828,"15min":199.828},"last":16.451000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":23.556999999999999,"5min":12.887,"15min":16.77},"min":{"1min":6.3620000000000001,"5min":6.234,"15min":5.8789999999999996},"max":{"1min":199.03800000000001,"5min":199.03800000000001,"15min":199.03800000000001},"last":9.1419999999999995},{"interface":"front","average":{"1min":26.257999999999999,"5min":19.350999999999999,"15min":19.757000000000001},"min":{"1min":11.548999999999999,"5min":11.548999999999999,"15min":6.1479999999999997},"max":{"1min":113.911,"5min":113.911,"15min":268.072},"last":14.778}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":22.385000000000002,"5min":19.629999999999999,"15min":22.059999999999999},"min":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":8.3239999999999998},"max":{"1min":63.686999999999998,"5min":63.686999999999998,"15min":198.304},"last":17.050000000000001},{"interface":"front","average":{"1min":19.053000000000001,"5min":15.726000000000001,"15min":20.466000000000001},"min":{"1min":10.156000000000001,"5min":8.0630000000000006,"15min":8.0630000000000006},"max":{"1min":56.078000000000003,"5min":56.078000000000003,"15min":181.84299999999999},"last":12.374000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-11T20:49:13.069 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.070 INFO:tasks.ceph:pgid 8.1a last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.070 INFO:tasks.ceph:pgid 8.1b last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.070 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.070 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.070 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.070 INFO:tasks.ceph:pgid 8.7 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.070 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.070 INFO:tasks.ceph:pgid 8.1 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 8.3 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-11T20:16:54.852869+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=16, tm_sec=54, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 8.9 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-11T20:16:54.852869+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=16, tm_sec=54, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 8.a last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-11T20:16:54.852869+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=16, tm_sec=54, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 8.b last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.071 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-11T20:16:54.852869+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=16, tm_sec=54, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-11T20:16:54.852869+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=16, tm_sec=54, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 8.e last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-11T20:16:54.852869+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=16, tm_sec=54, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 8.f last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-11T20:16:54.852869+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=16, tm_sec=54, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 8.c last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-11T20:16:54.852869+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=16, tm_sec=54, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 8.d last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.072 INFO:tasks.ceph:pgid 8.1f last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.1e last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.13 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.073 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.074 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.074 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:13.074 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-11T20:49:14.716 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:27.824 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:14.719 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:27.827 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:14.731 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:27.839 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:14.761 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:27.869 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:14.762 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:27.830 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:14.764 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:27.872 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:14.765 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:27.833 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:14.779 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:27.847 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:14.782 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:27.890 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.730 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:32.838 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.734 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:32.842 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.746 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:32.854 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.777 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:32.885 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.778 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:32.846 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.780 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:32.888 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.780 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:32.848 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.791 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:32.859 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:19.792 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:32.899 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:33.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-11T20:49:33.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.177+0000 7f040fea9640 10 monclient: get_monmap_and_config 2023-12-11T20:49:33.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.177+0000 7f040fea9640 10 monclient: build_initial_monmap 2023-12-11T20:49:33.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.177+0000 7f040fea9640 10 monclient: monmap: 2023-12-11T20:49:33.176 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:33.176 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:33.176 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:33.178068+0000 2023-12-11T20:49:33.176 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:33.178068+0000 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:33.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 1 Processor -- start 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 1 -- start start 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient: init 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0408071690 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0408145d30 0x7f040813eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient(hunting): picked mon.noname-a con 0x7f0408145d30 addr v2:172.21.15.86:3300/0 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f040813f0b0 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040806bc90 con 0x7f0408071690 2023-12-11T20:49:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0408070940 con 0x7f040813f0b0 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0408070760 con 0x7f0408145d30 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.178+0000 7f040fea9640 10 monclient(hunting): authenticate will time out at 3459.508049s 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.179+0000 7f040e446640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f040813f0b0 0x7f040814cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.86:55330/0 (socket says 172.21.15.86:55330) 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.179+0000 7f040e446640 1 -- 172.21.15.86:0/402522085 learned_addr learned my addr 172.21.15.86:0/402522085 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.179+0000 7f040d444640 1 --2- 172.21.15.86:0/402522085 >> v2:172.21.15.86:3300/0 conn(0x7f0408145d30 0x7f040813eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.180+0000 7f040d444640 10 monclient(hunting): get_auth_request con 0x7f0408145d30 auth_method 0 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.180+0000 7f040d444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.180+0000 7f040d444640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.180+0000 7f040d444640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.180+0000 7f040d444640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.180+0000 7f040d444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.180+0000 7f040d444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:33.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.181+0000 7f040cc43640 1 -- 172.21.15.86:0/402522085 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683683850 0 0) 0x7f0408070940 con 0x7f040813f0b0 2023-12-11T20:49:33.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.181+0000 7f040cc43640 10 monclient(hunting): _negotiate 2023-12-11T20:49:33.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.181+0000 7f040cc43640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:33.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.181+0000 7f040cc43640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.181+0000 7f040cc43640 10 monclient(hunting): my global_id is 21118 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.181+0000 7f040cc43640 1 -- 172.21.15.86:0/402522085 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f4003520 con 0x7f040813f0b0 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.182+0000 7f040d444640 10 monclient(hunting): handle_auth_done global_id 16962 payload 274 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.182+0000 7f040d444640 1 -- 172.21.15.86:0/402522085 >> v1:172.21.15.133:6789/0 conn(0x7f040813f0b0 legacy=0x7f040814cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.182+0000 7f040d444640 1 -- 172.21.15.86:0/402522085 >> v1:172.21.15.86:6789/0 conn(0x7f0408071690 legacy=0x7f0408145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.182+0000 7f040d444640 10 monclient: _finish_hunting 0 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.182+0000 7f040d444640 1 monclient: found mon.noname-a 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.182+0000 7f040d444640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.182+0000 7f040d444640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2023-12-11T20:49:33.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.182+0000 7f040d444640 1 -- 172.21.15.86:0/402522085 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0408070c20 con 0x7f0408145d30 2023-12-11T20:49:33.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.184+0000 7f040d444640 1 --2- 172.21.15.86:0/402522085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408145d30 0x7f040813eb70 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f03f80097a0 tx=0x7f03f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03774f390e3cef4 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 1 -- 172.21.15.86:0/402522085 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8031830 con 0x7f0408145d30 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: dump: 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: _finish_auth 0 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: _check_auth_tickets 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 1 -- 172.21.15.86:0/402522085 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f8031990 con 0x7f0408145d30 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 1 -- 172.21.15.86:0/402522085 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8031d20 con 0x7f0408145d30 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040fea9640 5 monclient: authenticate success, global_id 16962 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040fea9640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040fea9640 4 set_mon_vals no callback set 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.188+0000 7f040cc43640 10 monclient: dump: 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:33.189 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 10 monclient: shutdown 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 1 -- 172.21.15.86:0/402522085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408145d30 msgr2=0x7f040813eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 1 --2- 172.21.15.86:0/402522085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408145d30 0x7f040813eb70 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f03f80097a0 tx=0x7f03f802b8d0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 1 -- 172.21.15.86:0/402522085 shutdown_connections 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 1 --2- 172.21.15.86:0/402522085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408145d30 0x7f040813eb70 unknown :-1 s=CLOSED pgs=3365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 1 -- 172.21.15.86:0/402522085 >> 172.21.15.86:0/402522085 conn(0x7f04080b5830 msgr2=0x7f04080b5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 1 -- 172.21.15.86:0/402522085 shutdown_connections 2023-12-11T20:49:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.189+0000 7f040fea9640 1 -- 172.21.15.86:0/402522085 wait complete. 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 10 monclient: build_initial_monmap 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 10 monclient: monmap: 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 1 librados: starting msgr at 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 1 librados: starting objecter 2023-12-11T20:49:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 1 Processor -- start 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 1 -- start start 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 1 librados: setting wanted keys 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 1 librados: calling monclient init 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.190+0000 7f040fea9640 10 monclient: init 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408071690 0x7f0408142520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient(hunting): picked mon.noname-a con 0x7f0408071690 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f040813f0b0 0x7f04081d7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient(hunting): picked mon.noname-b con 0x7f040813f0b0 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0408145d30 0x7f04080acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient(hunting): picked mon.noname-c con 0x7f0408145d30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081d7de0 con 0x7f0408071690 2023-12-11T20:49:33.195 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081d7950 con 0x7f040813f0b0 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04081d7b10 con 0x7f0408145d30 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040fea9640 10 monclient(hunting): authenticate will time out at 3459.520328s 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040dc45640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408071690 0x7f0408142520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040d444640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f040813f0b0 0x7f04081d7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040e446640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0408145d30 0x7f04080acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040d444640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f040813f0b0 0x7f04081d7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:58132/0 (socket says 172.21.15.86:58132) 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040d444640 1 -- 172.21.15.86:0/1640881604 learned_addr learned my addr 172.21.15.86:0/1640881604 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040d444640 10 monclient(hunting): get_auth_request con 0x7f040813f0b0 auth_method 0 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040d444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040d444640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.191+0000 7f040d444640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:33.196 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040dc45640 10 monclient(hunting): get_auth_request con 0x7f0408071690 auth_method 0 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040dc45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040dc45640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040dc45640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040e446640 10 monclient(hunting): get_auth_request con 0x7f0408145d30 auth_method 0 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040e446640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040e446640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040e446640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040d444640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040d444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040d444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040dc45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040dc45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040dc45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040e446640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040e446640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.192+0000 7f040e446640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.194+0000 7f040d444640 10 monclient(hunting): handle_auth_done global_id 21121 payload 824 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.194+0000 7f040d444640 1 -- 172.21.15.86:0/1640881604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0408145d30 msgr2=0x7f04080acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.194+0000 7f040d444640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0408145d30 0x7f04080acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.194+0000 7f040d444640 1 -- 172.21.15.86:0/1640881604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408071690 msgr2=0x7f0408142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.194+0000 7f040d444640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408071690 0x7f0408142520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.194+0000 7f040d444640 10 monclient: _finish_hunting 0 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.194+0000 7f040d444640 1 monclient: found mon.noname-b 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.195+0000 7f040d444640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.195+0000 7f040d444640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.195+0000 7f040d444640 1 -- 172.21.15.86:0/1640881604 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f8009400 con 0x7f040813f0b0 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.197+0000 7f040d444640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f040813f0b0 0x7f04081d7410 secure :-1 s=READY pgs=5136 cs=0 l=1 rev1=1 crypto rx=0x7f03f80098c0 tx=0x7f03f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.197+0000 7f03fe7fc640 10 client.?.objecter ms_handle_connect 0x7f040813f0b0 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.197+0000 7f03fe7fc640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 1 -- 172.21.15.86:0/1640881604 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f803a8f0 con 0x7f040813f0b0 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: dump: 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: _finish_auth 0 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 1 -- 172.21.15.86:0/1640881604 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f803aa70 con 0x7f040813f0b0 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 1 -- 172.21.15.86:0/1640881604 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f803ae20 con 0x7f040813f0b0 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 5 monclient: authenticate success, global_id 21121 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 10 monclient: _renew_subs 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040813b5d0 con 0x7f040813f0b0 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fffff640 4 set_mon_vals no callback set 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f03fe7fc640 10 monclient: dump: 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 10 client.21121.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 10 monclient: _renew_subs 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04080ad230 con 0x7f040813f0b0 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.200+0000 7f040fea9640 1 librados: init done 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.201+0000 7f040fea9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.201+0000 7f040fea9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.201+0000 7f040fea9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.201+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04080ac8d0 con 0x7f040813f0b0 2023-12-11T20:49:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.440+0000 7f03fe7fc640 1 -- 172.21.15.86:0/1640881604 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f03f804d450 con 0x7f040813f0b0 2023-12-11T20:49:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.441+0000 7f03fe7fc640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f03ec06c870 0x7f03ec06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.441+0000 7f040dc45640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f03ec06c870 0x7f03ec06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.441+0000 7f040dc45640 10 monclient: get_auth_request con 0x7f03ec06c870 auth_method 0 2023-12-11T20:49:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.441+0000 7f040dc45640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f03ec06c870 0x7f03ec06ed30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f04040017f0 tx=0x7f0404009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.441+0000 7f03fe7fc640 10 client.21121.objecter ms_handle_connect 0x7f03ec06c870 2023-12-11T20:49:33.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.444+0000 7f03fe7fc640 1 -- 172.21.15.86:0/1640881604 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f03f807a7c0 con 0x7f040813f0b0 2023-12-11T20:49:33.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.444+0000 7f03fe7fc640 10 client.21121.objecter ms_dispatch 0x7f0408000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:33.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.444+0000 7f03fe7fc640 3 client.21121.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:33.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.444+0000 7f03fe7fc640 3 client.21121.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:33.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.444+0000 7f03fe7fc640 20 client.21121.objecter dump_active .. 0 homeless 2023-12-11T20:49:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.453+0000 7f03fe7fc640 1 -- 172.21.15.86:0/1640881604 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03f8076b00 con 0x7f040813f0b0 2023-12-11T20:49:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.453+0000 7f03fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.453+0000 7f03fe7fc640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:33.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.619+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f040806bee0 con 0x7f03ec06c870 2023-12-11T20:49:33.623 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.625+0000 7f03fe7fc640 1 -- 172.21.15.86:0/1640881604 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151763 (secure 0 0 0) 0x7f0404001440 con 0x7f03ec06c870 2023-12-11T20:49:33.625 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:49:33.626 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-11T20:49:33.627 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 10 librados: watch_flush enter 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 10 librados: watch_flush exit 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 20 client.21121.objecter shutdown clearing up homeless session... 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 10 client.21121.objecter successfully canceled tick 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f03ec06c870 msgr2=0x7f03ec06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f03ec06c870 0x7f03ec06ed30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f04040017f0 tx=0x7f0404009040 comp rx=0 tx=0).stop 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 10 monclient: shutdown 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f040813f0b0 msgr2=0x7f04081d7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f040813f0b0 0x7f04081d7410 secure :-1 s=READY pgs=5136 cs=0 l=1 rev1=1 crypto rx=0x7f03f80098c0 tx=0x7f03f80046c0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 shutdown_connections 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f03ec06c870 0x7f03ec06ed30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0408145d30 0x7f04080acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f040813f0b0 0x7f04081d7410 unknown :-1 s=CLOSED pgs=5136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 --2- 172.21.15.86:0/1640881604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0408071690 0x7f0408142520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 >> 172.21.15.86:0/1640881604 conn(0x7f04080b5830 msgr2=0x7f040813e200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:33.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 shutdown_connections 2023-12-11T20:49:33.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 -- 172.21.15.86:0/1640881604 wait complete. 2023-12-11T20:49:33.629 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:33.627+0000 7f040fea9640 1 librados: shutdown 2023-12-11T20:49:33.637 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":2045,"stamp":"2023-12-11T20:49:33.116136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":534584,"kb_used_data":17208,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754440136,"statfs":{"total":773094113280,"available":772546699264,"internally_reserved":0,"allocated":17620992,"data_stored":5245138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":529714737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003080"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1012,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:07.141741+0000","last_change":"2023-12-11T20:49:07.141741+0000","last_active":"2023-12-11T20:49:07.141741+0000","last_peered":"2023-12-11T20:49:07.141741+0000","last_clean":"2023-12-11T20:49:07.141741+0000","last_became_active":"2023-12-11T20:18:00.530681+0000","last_became_peered":"2023-12-11T20:18:00.530681+0000","last_unstale":"2023-12-11T20:49:07.141741+0000","last_undegraded":"2023-12-11T20:49:07.141741+0000","last_fullsized":"2023-12-11T20:49:07.141741+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:07.138431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:07.138431+0000","last_clean_scrub_stamp":"2023-12-11T20:49:07.138431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:43:34.623842+0000","scrub_duration":0.031828385000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:10.009088+0000","last_change":"2023-12-11T20:49:10.009088+0000","last_active":"2023-12-11T20:49:10.009088+0000","last_peered":"2023-12-11T20:49:10.009088+0000","last_clean":"2023-12-11T20:49:10.009088+0000","last_became_active":"2023-12-11T20:18:01.459740+0000","last_became_peered":"2023-12-11T20:18:01.459740+0000","last_unstale":"2023-12-11T20:49:10.009088+0000","last_undegraded":"2023-12-11T20:49:10.009088+0000","last_fullsized":"2023-12-11T20:49:10.009088+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:10.006984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:10.006984+0000","last_clean_scrub_stamp":"2023-12-11T20:49:10.006984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:40:50.627346+0000","scrub_duration":0.184697679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1020,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:14.479786+0000","last_change":"2023-12-11T20:49:14.480468+0000","last_active":"2023-12-11T20:49:14.479786+0000","last_peered":"2023-12-11T20:49:14.479786+0000","last_clean":"2023-12-11T20:49:14.479786+0000","last_became_active":"2023-12-11T20:18:11.552993+0000","last_became_peered":"2023-12-11T20:18:11.552993+0000","last_unstale":"2023-12-11T20:49:14.479786+0000","last_undegraded":"2023-12-11T20:49:14.479786+0000","last_fullsized":"2023-12-11T20:49:14.479786+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:14.478434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:14.478434+0000","last_clean_scrub_stamp":"2023-12-11T20:49:14.478434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:44:01.670384+0000","scrub_duration":0.044903595999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1015,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:18.955364+0000","last_change":"2023-12-11T20:49:18.956011+0000","last_active":"2023-12-11T20:49:18.955364+0000","last_peered":"2023-12-11T20:49:18.955364+0000","last_clean":"2023-12-11T20:49:18.955364+0000","last_became_active":"2023-12-11T20:18:04.748213+0000","last_became_peered":"2023-12-11T20:18:04.748213+0000","last_unstale":"2023-12-11T20:49:18.955364+0000","last_undegraded":"2023-12-11T20:49:18.955364+0000","last_fullsized":"2023-12-11T20:49:18.955364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:18.954003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:18.954003+0000","last_clean_scrub_stamp":"2023-12-11T20:49:18.954003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:57:36.982193+0000","scrub_duration":0.053313004999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:13.889915+0000","last_change":"2023-12-11T20:49:13.889915+0000","last_active":"2023-12-11T20:49:13.889915+0000","last_peered":"2023-12-11T20:49:13.889915+0000","last_clean":"2023-12-11T20:49:13.889915+0000","last_became_active":"2023-12-11T20:18:06.871380+0000","last_became_peered":"2023-12-11T20:18:06.871380+0000","last_unstale":"2023-12-11T20:49:13.889915+0000","last_undegraded":"2023-12-11T20:49:13.889915+0000","last_fullsized":"2023-12-11T20:49:13.889915+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:13.887611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:13.887611+0000","last_clean_scrub_stamp":"2023-12-11T20:49:13.887611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:26:11.335636+0000","scrub_duration":0.023335913,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1021,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:13.931187+0000","last_change":"2023-12-11T20:49:13.931187+0000","last_active":"2023-12-11T20:49:13.931187+0000","last_peered":"2023-12-11T20:49:13.931187+0000","last_clean":"2023-12-11T20:49:13.931187+0000","last_became_active":"2023-12-11T20:18:00.891399+0000","last_became_peered":"2023-12-11T20:18:00.891399+0000","last_unstale":"2023-12-11T20:49:13.931187+0000","last_undegraded":"2023-12-11T20:49:13.931187+0000","last_fullsized":"2023-12-11T20:49:13.931187+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:13.929236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:13.929236+0000","last_clean_scrub_stamp":"2023-12-11T20:49:13.929236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:15:44.597307+0000","scrub_duration":0.056576538000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1022,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:13.450413+0000","last_change":"2023-12-11T20:49:13.451067+0000","last_active":"2023-12-11T20:49:13.450413+0000","last_peered":"2023-12-11T20:49:13.450413+0000","last_clean":"2023-12-11T20:49:13.450413+0000","last_became_active":"2023-12-11T20:18:11.323743+0000","last_became_peered":"2023-12-11T20:18:11.323743+0000","last_unstale":"2023-12-11T20:49:13.450413+0000","last_undegraded":"2023-12-11T20:49:13.450413+0000","last_fullsized":"2023-12-11T20:49:13.450413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:13.449047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:13.449047+0000","last_clean_scrub_stamp":"2023-12-11T20:49:13.449047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:01:48.607227+0000","scrub_duration":0.045758721000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1033,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:27.416519+0000","last_change":"2023-12-11T20:49:27.417180+0000","last_active":"2023-12-11T20:49:27.416519+0000","last_peered":"2023-12-11T20:49:27.416519+0000","last_clean":"2023-12-11T20:49:27.416519+0000","last_became_active":"2023-12-11T20:18:03.798156+0000","last_became_peered":"2023-12-11T20:18:03.798156+0000","last_unstale":"2023-12-11T20:49:27.416519+0000","last_undegraded":"2023-12-11T20:49:27.416519+0000","last_fullsized":"2023-12-11T20:49:27.416519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:27.415205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:27.415205+0000","last_clean_scrub_stamp":"2023-12-11T20:49:27.415205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T00:01:34.614827+0000","scrub_duration":0.073947849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1026,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:12.947697+0000","last_change":"2023-12-11T20:49:12.948332+0000","last_active":"2023-12-11T20:49:12.947697+0000","last_peered":"2023-12-11T20:49:12.947697+0000","last_clean":"2023-12-11T20:49:12.947697+0000","last_became_active":"2023-12-11T20:18:06.509738+0000","last_became_peered":"2023-12-11T20:18:06.509738+0000","last_unstale":"2023-12-11T20:49:12.947697+0000","last_undegraded":"2023-12-11T20:49:12.947697+0000","last_fullsized":"2023-12-11T20:49:12.947697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:12.946399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:12.946399+0000","last_clean_scrub_stamp":"2023-12-11T20:49:12.946399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:44:02.649287+0000","scrub_duration":0.023431917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"1057'8","reported_seq":1037,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:21.373637+0000","last_change":"2023-12-11T20:49:21.374468+0000","last_active":"2023-12-11T20:49:21.373637+0000","last_peered":"2023-12-11T20:49:21.373637+0000","last_clean":"2023-12-11T20:49:21.373637+0000","last_became_active":"2023-12-11T20:18:05.684774+0000","last_became_peered":"2023-12-11T20:18:05.684774+0000","last_unstale":"2023-12-11T20:49:21.373637+0000","last_undegraded":"2023-12-11T20:49:21.373637+0000","last_fullsized":"2023-12-11T20:49:21.373637+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"1057'8","last_scrub_stamp":"2023-12-11T20:49:21.372297+0000","last_deep_scrub":"1057'8","last_deep_scrub_stamp":"2023-12-11T20:49:21.372297+0000","last_clean_scrub_stamp":"2023-12-11T20:49:21.372297+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:45:17.751307+0000","scrub_duration":0.080219269999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:09.868318+0000","last_change":"2023-12-11T20:49:09.868318+0000","last_active":"2023-12-11T20:49:09.868318+0000","last_peered":"2023-12-11T20:49:09.868318+0000","last_clean":"2023-12-11T20:49:09.868318+0000","last_became_active":"2023-12-11T20:18:11.576567+0000","last_became_peered":"2023-12-11T20:18:11.576567+0000","last_unstale":"2023-12-11T20:49:09.868318+0000","last_undegraded":"2023-12-11T20:49:09.868318+0000","last_fullsized":"2023-12-11T20:49:09.868318+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:09.866209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:09.866209+0000","last_clean_scrub_stamp":"2023-12-11T20:49:09.866209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:07:41.978437+0000","scrub_duration":0.034387038000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1034,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:20.485168+0000","last_change":"2023-12-11T20:49:20.485805+0000","last_active":"2023-12-11T20:49:20.485168+0000","last_peered":"2023-12-11T20:49:20.485168+0000","last_clean":"2023-12-11T20:49:20.485168+0000","last_became_active":"2023-12-11T20:18:03.237738+0000","last_became_peered":"2023-12-11T20:18:03.237738+0000","last_unstale":"2023-12-11T20:49:20.485168+0000","last_undegraded":"2023-12-11T20:49:20.485168+0000","last_fullsized":"2023-12-11T20:49:20.485168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:20.483814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:20.483814+0000","last_clean_scrub_stamp":"2023-12-11T20:49:20.483814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:41:56.135136+0000","scrub_duration":0.065074221000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1057'78","reported_seq":1133,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:08.936824+0000","last_change":"2023-12-11T20:49:08.937802+0000","last_active":"2023-12-11T20:49:08.936824+0000","last_peered":"2023-12-11T20:49:08.936824+0000","last_clean":"2023-12-11T20:49:08.936824+0000","last_became_active":"2023-12-11T20:16:49.435576+0000","last_became_peered":"2023-12-11T20:16:49.435576+0000","last_unstale":"2023-12-11T20:49:08.936824+0000","last_undegraded":"2023-12-11T20:49:08.936824+0000","last_fullsized":"2023-12-11T20:49:08.936824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"1057'78","last_scrub_stamp":"2023-12-11T20:49:08.935520+0000","last_deep_scrub":"1057'78","last_deep_scrub_stamp":"2023-12-11T20:49:08.935520+0000","last_clean_scrub_stamp":"2023-12-11T20:49:08.935520+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:34:51.152246+0000","scrub_duration":0.11416854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:11.079430+0000","last_change":"2023-12-11T20:49:11.080091+0000","last_active":"2023-12-11T20:49:11.079430+0000","last_peered":"2023-12-11T20:49:11.079430+0000","last_clean":"2023-12-11T20:49:11.079430+0000","last_became_active":"2023-12-11T20:16:58.638544+0000","last_became_peered":"2023-12-11T20:16:58.638544+0000","last_unstale":"2023-12-11T20:49:11.079430+0000","last_undegraded":"2023-12-11T20:49:11.079430+0000","last_fullsized":"2023-12-11T20:49:11.079430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:11.078127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:11.078127+0000","last_clean_scrub_stamp":"2023-12-11T20:49:11.078127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:20:31.188162+0000","scrub_duration":0.025714267999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1013,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:14.181425+0000","last_change":"2023-12-11T20:49:14.182122+0000","last_active":"2023-12-11T20:49:14.181425+0000","last_peered":"2023-12-11T20:49:14.181425+0000","last_clean":"2023-12-11T20:49:14.181425+0000","last_became_active":"2023-12-11T20:18:12.924715+0000","last_became_peered":"2023-12-11T20:18:12.924715+0000","last_unstale":"2023-12-11T20:49:14.181425+0000","last_undegraded":"2023-12-11T20:49:14.181425+0000","last_fullsized":"2023-12-11T20:49:14.181425+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:14.180019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:14.180019+0000","last_clean_scrub_stamp":"2023-12-11T20:49:14.180019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:16:50.154388+0000","scrub_duration":0.054111566999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:10.334120+0000","last_change":"2023-12-11T20:49:10.338763+0000","last_active":"2023-12-11T20:49:10.334120+0000","last_peered":"2023-12-11T20:49:10.334120+0000","last_clean":"2023-12-11T20:49:10.334120+0000","last_became_active":"2023-12-11T20:16:58.008642+0000","last_became_peered":"2023-12-11T20:16:58.008642+0000","last_unstale":"2023-12-11T20:49:10.334120+0000","last_undegraded":"2023-12-11T20:49:10.334120+0000","last_fullsized":"2023-12-11T20:49:10.334120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:10.330090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:10.330090+0000","last_clean_scrub_stamp":"2023-12-11T20:49:10.330090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:05:06.782303+0000","scrub_duration":0.21427059700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1026,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:22.961515+0000","last_change":"2023-12-11T20:49:22.962187+0000","last_active":"2023-12-11T20:49:22.961515+0000","last_peered":"2023-12-11T20:49:22.961515+0000","last_clean":"2023-12-11T20:49:22.961515+0000","last_became_active":"2023-12-11T20:18:07.135656+0000","last_became_peered":"2023-12-11T20:18:07.135656+0000","last_unstale":"2023-12-11T20:49:22.961515+0000","last_undegraded":"2023-12-11T20:49:22.961515+0000","last_fullsized":"2023-12-11T20:49:22.961515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:22.960080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:22.960080+0000","last_clean_scrub_stamp":"2023-12-11T20:49:22.960080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:41:09.238717+0000","scrub_duration":0.029403362999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1038,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:13.509279+0000","last_change":"2023-12-11T20:49:13.509933+0000","last_active":"2023-12-11T20:49:13.509279+0000","last_peered":"2023-12-11T20:49:13.509279+0000","last_clean":"2023-12-11T20:49:13.509279+0000","last_became_active":"2023-12-11T20:16:58.208987+0000","last_became_peered":"2023-12-11T20:16:58.208987+0000","last_unstale":"2023-12-11T20:49:13.509279+0000","last_undegraded":"2023-12-11T20:49:13.509279+0000","last_fullsized":"2023-12-11T20:49:13.509279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:13.507891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:13.507891+0000","last_clean_scrub_stamp":"2023-12-11T20:49:13.507891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:18:30.657122+0000","scrub_duration":0.026209349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1034,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:15.379092+0000","last_change":"2023-12-11T20:49:15.379092+0000","last_active":"2023-12-11T20:49:15.379092+0000","last_peered":"2023-12-11T20:49:15.379092+0000","last_clean":"2023-12-11T20:49:15.379092+0000","last_became_active":"2023-12-11T20:18:01.354083+0000","last_became_peered":"2023-12-11T20:18:01.354083+0000","last_unstale":"2023-12-11T20:49:15.379092+0000","last_undegraded":"2023-12-11T20:49:15.379092+0000","last_fullsized":"2023-12-11T20:49:15.379092+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:15.377081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:15.377081+0000","last_clean_scrub_stamp":"2023-12-11T20:49:15.377081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:10:33.165194+0000","scrub_duration":0.080873467000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:18.070420+0000","last_change":"2023-12-11T20:49:18.071089+0000","last_active":"2023-12-11T20:49:18.070420+0000","last_peered":"2023-12-11T20:49:18.070420+0000","last_clean":"2023-12-11T20:49:18.070420+0000","last_became_active":"2023-12-11T20:16:58.349599+0000","last_became_peered":"2023-12-11T20:16:58.349599+0000","last_unstale":"2023-12-11T20:49:18.070420+0000","last_undegraded":"2023-12-11T20:49:18.070420+0000","last_fullsized":"2023-12-11T20:49:18.070420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:18.069072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:18.069072+0000","last_clean_scrub_stamp":"2023-12-11T20:49:18.069072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:26:02.233542+0000","scrub_duration":0.025866693999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:10.837573+0000","last_change":"2023-12-11T20:49:10.838821+0000","last_active":"2023-12-11T20:49:10.837573+0000","last_peered":"2023-12-11T20:49:10.837573+0000","last_clean":"2023-12-11T20:49:10.837573+0000","last_became_active":"2023-12-11T20:18:02.117789+0000","last_became_peered":"2023-12-11T20:18:02.117789+0000","last_unstale":"2023-12-11T20:49:10.837573+0000","last_undegraded":"2023-12-11T20:49:10.837573+0000","last_fullsized":"2023-12-11T20:49:10.837573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:10.835332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:10.835332+0000","last_clean_scrub_stamp":"2023-12-11T20:49:10.835332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:14:03.415524+0000","scrub_duration":0.045176734000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1031,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:10.958036+0000","last_change":"2023-12-11T20:49:10.958716+0000","last_active":"2023-12-11T20:49:10.958036+0000","last_peered":"2023-12-11T20:49:10.958036+0000","last_clean":"2023-12-11T20:49:10.958036+0000","last_became_active":"2023-12-11T20:16:58.383357+0000","last_became_peered":"2023-12-11T20:16:58.383357+0000","last_unstale":"2023-12-11T20:49:10.958036+0000","last_undegraded":"2023-12-11T20:49:10.958036+0000","last_fullsized":"2023-12-11T20:49:10.958036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:10.955322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:10.955322+0000","last_clean_scrub_stamp":"2023-12-11T20:49:10.955322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:37:08.717834+0000","scrub_duration":0.026970309000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1017,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.483669+0000","last_change":"2023-12-11T20:18:13.718317+0000","last_active":"2023-12-11T20:49:02.483669+0000","last_peered":"2023-12-11T20:49:02.483669+0000","last_clean":"2023-12-11T20:49:02.483669+0000","last_became_active":"2023-12-11T20:18:13.700424+0000","last_became_peered":"2023-12-11T20:18:13.700424+0000","last_unstale":"2023-12-11T20:49:02.483669+0000","last_undegraded":"2023-12-11T20:49:02.483669+0000","last_fullsized":"2023-12-11T20:49:02.483669+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:23.074066+0000","last_change":"2023-12-11T20:49:23.074747+0000","last_active":"2023-12-11T20:49:23.074066+0000","last_peered":"2023-12-11T20:49:23.074066+0000","last_clean":"2023-12-11T20:49:23.074066+0000","last_became_active":"2023-12-11T20:16:58.674889+0000","last_became_peered":"2023-12-11T20:16:58.674889+0000","last_unstale":"2023-12-11T20:49:23.074066+0000","last_undegraded":"2023-12-11T20:49:23.074066+0000","last_fullsized":"2023-12-11T20:49:23.074066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:23.072765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:23.072765+0000","last_clean_scrub_stamp":"2023-12-11T20:49:23.072765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:37:27.674302+0000","scrub_duration":0.024438919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1033,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:24.456293+0000","last_change":"2023-12-11T20:49:24.456976+0000","last_active":"2023-12-11T20:49:24.456293+0000","last_peered":"2023-12-11T20:49:24.456293+0000","last_clean":"2023-12-11T20:49:24.456293+0000","last_became_active":"2023-12-11T20:18:03.180691+0000","last_became_peered":"2023-12-11T20:18:03.180691+0000","last_unstale":"2023-12-11T20:49:24.456293+0000","last_undegraded":"2023-12-11T20:49:24.456293+0000","last_fullsized":"2023-12-11T20:49:24.456293+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:24.454965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:24.454965+0000","last_clean_scrub_stamp":"2023-12-11T20:49:24.454965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:36:06.006630+0000","scrub_duration":0.080620827000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1039,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:11.305527+0000","last_change":"2023-12-11T20:49:11.305527+0000","last_active":"2023-12-11T20:49:11.305527+0000","last_peered":"2023-12-11T20:49:11.305527+0000","last_clean":"2023-12-11T20:49:11.305527+0000","last_became_active":"2023-12-11T20:16:57.255904+0000","last_became_peered":"2023-12-11T20:16:57.255904+0000","last_unstale":"2023-12-11T20:49:11.305527+0000","last_undegraded":"2023-12-11T20:49:11.305527+0000","last_fullsized":"2023-12-11T20:49:11.305527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:11.303567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:11.303567+0000","last_clean_scrub_stamp":"2023-12-11T20:49:11.303567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:47:39.399616+0000","scrub_duration":0.024613408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1025,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:23.949895+0000","last_change":"2023-12-11T20:49:23.950534+0000","last_active":"2023-12-11T20:49:23.949895+0000","last_peered":"2023-12-11T20:49:23.949895+0000","last_clean":"2023-12-11T20:49:23.949895+0000","last_became_active":"2023-12-11T20:18:33.762733+0000","last_became_peered":"2023-12-11T20:18:33.762733+0000","last_unstale":"2023-12-11T20:49:23.949895+0000","last_undegraded":"2023-12-11T20:49:23.949895+0000","last_fullsized":"2023-12-11T20:49:23.949895+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:23.948571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:23.948571+0000","last_clean_scrub_stamp":"2023-12-11T20:49:23.948571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:11:37.564191+0000","scrub_duration":0.028664346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:14.121459+0000","last_change":"2023-12-11T20:49:14.121459+0000","last_active":"2023-12-11T20:49:14.121459+0000","last_peered":"2023-12-11T20:49:14.121459+0000","last_clean":"2023-12-11T20:49:14.121459+0000","last_became_active":"2023-12-11T20:16:58.071647+0000","last_became_peered":"2023-12-11T20:16:58.071647+0000","last_unstale":"2023-12-11T20:49:14.121459+0000","last_undegraded":"2023-12-11T20:49:14.121459+0000","last_fullsized":"2023-12-11T20:49:14.121459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:14.118031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:14.118031+0000","last_clean_scrub_stamp":"2023-12-11T20:49:14.118031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:10:43.276760+0000","scrub_duration":0.031451593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1016,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:19.970059+0000","last_change":"2023-12-11T20:49:19.970759+0000","last_active":"2023-12-11T20:49:19.970059+0000","last_peered":"2023-12-11T20:49:19.970059+0000","last_clean":"2023-12-11T20:49:19.970059+0000","last_became_active":"2023-12-11T20:18:11.885388+0000","last_became_peered":"2023-12-11T20:18:11.885388+0000","last_unstale":"2023-12-11T20:49:19.970059+0000","last_undegraded":"2023-12-11T20:49:19.970059+0000","last_fullsized":"2023-12-11T20:49:19.970059+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:19.968634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:19.968634+0000","last_clean_scrub_stamp":"2023-12-11T20:49:19.968634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T00:21:57.498718+0000","scrub_duration":0.073009713000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1035,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:17.408654+0000","last_change":"2023-12-11T20:49:17.409350+0000","last_active":"2023-12-11T20:49:17.408654+0000","last_peered":"2023-12-11T20:49:17.408654+0000","last_clean":"2023-12-11T20:49:17.408654+0000","last_became_active":"2023-12-11T20:18:12.786107+0000","last_became_peered":"2023-12-11T20:18:12.786107+0000","last_unstale":"2023-12-11T20:49:17.408654+0000","last_undegraded":"2023-12-11T20:49:17.408654+0000","last_fullsized":"2023-12-11T20:49:17.408654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:17.407244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:17.407244+0000","last_clean_scrub_stamp":"2023-12-11T20:49:17.407244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:05:03.190257+0000","scrub_duration":0.14288842400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1034,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:23.447118+0000","last_change":"2023-12-11T20:49:23.447754+0000","last_active":"2023-12-11T20:49:23.447118+0000","last_peered":"2023-12-11T20:49:23.447118+0000","last_clean":"2023-12-11T20:49:23.447118+0000","last_became_active":"2023-12-11T20:18:00.469176+0000","last_became_peered":"2023-12-11T20:18:00.469176+0000","last_unstale":"2023-12-11T20:49:23.447118+0000","last_undegraded":"2023-12-11T20:49:23.447118+0000","last_fullsized":"2023-12-11T20:49:23.447118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:23.445559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:23.445559+0000","last_clean_scrub_stamp":"2023-12-11T20:49:23.445559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:32:20.460753+0000","scrub_duration":0.041653294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1020,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:15.475070+0000","last_change":"2023-12-11T20:49:15.475942+0000","last_active":"2023-12-11T20:49:15.475070+0000","last_peered":"2023-12-11T20:49:15.475070+0000","last_clean":"2023-12-11T20:49:15.475070+0000","last_became_active":"2023-12-11T20:18:05.138035+0000","last_became_peered":"2023-12-11T20:18:05.138035+0000","last_unstale":"2023-12-11T20:49:15.475070+0000","last_undegraded":"2023-12-11T20:49:15.475070+0000","last_fullsized":"2023-12-11T20:49:15.475070+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:15.473598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:15.473598+0000","last_clean_scrub_stamp":"2023-12-11T20:49:15.473598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:05:17.067244+0000","scrub_duration":0.049786271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1008,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:02.004145+0000","last_change":"2023-12-11T20:18:10.100025+0000","last_active":"2023-12-11T20:49:02.004145+0000","last_peered":"2023-12-11T20:49:02.004145+0000","last_clean":"2023-12-11T20:49:02.004145+0000","last_became_active":"2023-12-11T20:18:10.094694+0000","last_became_peered":"2023-12-11T20:18:10.094694+0000","last_unstale":"2023-12-11T20:49:02.004145+0000","last_undegraded":"2023-12-11T20:49:02.004145+0000","last_fullsized":"2023-12-11T20:49:02.004145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1024,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:14.993451+0000","last_change":"2023-12-11T20:49:14.994122+0000","last_active":"2023-12-11T20:49:14.993451+0000","last_peered":"2023-12-11T20:49:14.993451+0000","last_clean":"2023-12-11T20:49:14.993451+0000","last_became_active":"2023-12-11T20:18:07.062338+0000","last_became_peered":"2023-12-11T20:18:07.062338+0000","last_unstale":"2023-12-11T20:49:14.993451+0000","last_undegraded":"2023-12-11T20:49:14.993451+0000","last_fullsized":"2023-12-11T20:49:14.993451+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:14.992072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:14.992072+0000","last_clean_scrub_stamp":"2023-12-11T20:49:14.992072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:44:46.471306+0000","scrub_duration":0.089570364999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1009,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:01.993026+0000","last_change":"2023-12-11T20:18:19.678310+0000","last_active":"2023-12-11T20:49:01.993026+0000","last_peered":"2023-12-11T20:49:01.993026+0000","last_clean":"2023-12-11T20:49:01.993026+0000","last_became_active":"2023-12-11T20:18:19.665316+0000","last_became_peered":"2023-12-11T20:18:19.665316+0000","last_unstale":"2023-12-11T20:49:01.993026+0000","last_undegraded":"2023-12-11T20:49:01.993026+0000","last_fullsized":"2023-12-11T20:49:01.993026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1017,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:26.176549+0000","last_change":"2023-12-11T20:49:26.177232+0000","last_active":"2023-12-11T20:49:26.176549+0000","last_peered":"2023-12-11T20:49:26.176549+0000","last_clean":"2023-12-11T20:49:26.176549+0000","last_became_active":"2023-12-11T20:18:18.557249+0000","last_became_peered":"2023-12-11T20:18:18.557249+0000","last_unstale":"2023-12-11T20:49:26.176549+0000","last_undegraded":"2023-12-11T20:49:26.176549+0000","last_fullsized":"2023-12-11T20:49:26.176549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:26.175055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:26.175055+0000","last_clean_scrub_stamp":"2023-12-11T20:49:26.175055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:28:10.873713+0000","scrub_duration":0.063818097000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1016,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:25.100719+0000","last_change":"2023-12-11T20:49:25.101410+0000","last_active":"2023-12-11T20:49:25.100719+0000","last_peered":"2023-12-11T20:49:25.100719+0000","last_clean":"2023-12-11T20:49:25.100719+0000","last_became_active":"2023-12-11T20:18:19.685187+0000","last_became_peered":"2023-12-11T20:18:19.685187+0000","last_unstale":"2023-12-11T20:49:25.100719+0000","last_undegraded":"2023-12-11T20:49:25.100719+0000","last_fullsized":"2023-12-11T20:49:25.100719+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:25.099287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:25.099287+0000","last_clean_scrub_stamp":"2023-12-11T20:49:25.099287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:41:03.663936+0000","scrub_duration":0.029382286000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:15.998742+0000","last_change":"2023-12-11T20:49:15.998742+0000","last_active":"2023-12-11T20:49:15.998742+0000","last_peered":"2023-12-11T20:49:15.998742+0000","last_clean":"2023-12-11T20:49:15.998742+0000","last_became_active":"2023-12-11T20:18:00.554682+0000","last_became_peered":"2023-12-11T20:18:00.554682+0000","last_unstale":"2023-12-11T20:49:15.998742+0000","last_undegraded":"2023-12-11T20:49:15.998742+0000","last_fullsized":"2023-12-11T20:49:15.998742+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:15.996775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:15.996775+0000","last_clean_scrub_stamp":"2023-12-11T20:49:15.996775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:54:56.936253+0000","scrub_duration":0.075660405999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1025,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:30.042353+0000","last_change":"2023-12-11T20:49:30.042353+0000","last_active":"2023-12-11T20:49:30.042353+0000","last_peered":"2023-12-11T20:49:30.042353+0000","last_clean":"2023-12-11T20:49:30.042353+0000","last_became_active":"2023-12-11T20:18:13.689623+0000","last_became_peered":"2023-12-11T20:18:13.689623+0000","last_unstale":"2023-12-11T20:49:30.042353+0000","last_undegraded":"2023-12-11T20:49:30.042353+0000","last_fullsized":"2023-12-11T20:49:30.042353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:30.040447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:30.040447+0000","last_clean_scrub_stamp":"2023-12-11T20:49:30.040447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:57:12.232772+0000","scrub_duration":0.043914570999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1009,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:09.224961+0000","last_change":"2023-12-11T20:18:10.128111+0000","last_active":"2023-12-11T20:49:09.224961+0000","last_peered":"2023-12-11T20:49:09.224961+0000","last_clean":"2023-12-11T20:49:09.224961+0000","last_became_active":"2023-12-11T20:18:10.112911+0000","last_became_peered":"2023-12-11T20:18:10.112911+0000","last_unstale":"2023-12-11T20:49:09.224961+0000","last_undegraded":"2023-12-11T20:49:09.224961+0000","last_fullsized":"2023-12-11T20:49:09.224961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:17:56.659735+0000","last_clean_scrub_stamp":"2023-12-11T20:17:56.659735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1011,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:11.152306+0000","last_change":"2023-12-11T20:49:11.153004+0000","last_active":"2023-12-11T20:49:11.152306+0000","last_peered":"2023-12-11T20:49:11.152306+0000","last_clean":"2023-12-11T20:49:11.152306+0000","last_became_active":"2023-12-11T20:18:13.203517+0000","last_became_peered":"2023-12-11T20:18:13.203517+0000","last_unstale":"2023-12-11T20:49:11.152306+0000","last_undegraded":"2023-12-11T20:49:11.152306+0000","last_fullsized":"2023-12-11T20:49:11.152306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:11.150943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:11.150943+0000","last_clean_scrub_stamp":"2023-12-11T20:49:11.150943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:52:36.723162+0000","scrub_duration":0.026594514999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719477129,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69328,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302512,"statfs":{"total":96636764160,"available":96565772288,"internally_reserved":0,"allocated":2105344,"data_stored":556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":17.427,"5min":10.535,"15min":14.893000000000001},"min":{"1min":4.5739999999999998,"5min":4.5739999999999998,"15min":4.2510000000000003},"max":{"1min":112.848,"5min":112.848,"15min":221.21299999999999},"last":8.9420000000000002},{"interface":"front","average":{"1min":17.669,"5min":11.170999999999999,"15min":14.507999999999999},"min":{"1min":6.2409999999999997,"5min":5.3739999999999997,"15min":4.7930000000000001},"max":{"1min":94.400999999999996,"5min":94.400999999999996,"15min":164.01900000000001},"last":10.939}]},{"osd":1,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":21.391999999999999,"5min":12.878,"15min":16.920999999999999},"min":{"1min":7.6660000000000004,"5min":6.8419999999999996,"15min":5.0380000000000003},"max":{"1min":117.63,"5min":117.63,"15min":195.87200000000001},"last":6.6980000000000004},{"interface":"front","average":{"1min":26.209,"5min":17.108000000000001,"15min":20.576000000000001},"min":{"1min":10.843999999999999,"5min":9.6649999999999991,"15min":6.0300000000000002},"max":{"1min":118.309,"5min":118.309,"15min":190.53899999999999},"last":13.516999999999999}]},{"osd":2,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":15.789999999999999,"5min":11.457000000000001,"15min":18.841999999999999},"min":{"1min":6.4089999999999998,"5min":6.4089999999999998,"15min":5.8259999999999996},"max":{"1min":67.793000000000006,"5min":67.793000000000006,"15min":282.79000000000002},"last":7.3220000000000001},{"interface":"front","average":{"1min":21.603000000000002,"5min":15.961,"15min":21.614000000000001},"min":{"1min":8.1829999999999998,"5min":8.1829999999999998,"15min":8.0570000000000004},"max":{"1min":68.638999999999996,"5min":68.638999999999996,"15min":200.148},"last":17.302}]},{"osd":3,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.715,"5min":15.484999999999999,"15min":19.199999999999999},"min":{"1min":9.6660000000000004,"5min":6.5270000000000001,"15min":5.3029999999999999},"max":{"1min":113.89400000000001,"5min":113.89400000000001,"15min":206.11799999999999},"last":9.7699999999999996},{"interface":"front","average":{"1min":22.518000000000001,"5min":14.561,"15min":20.105},"min":{"1min":6.7809999999999997,"5min":6.7809999999999997,"15min":6.7809999999999997},"max":{"1min":140.172,"5min":140.172,"15min":171.63},"last":14.218}]},{"osd":4,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":24.693000000000001,"5min":16.344999999999999,"15min":18.094000000000001},"min":{"1min":8.5180000000000007,"5min":7.6289999999999996,"15min":6.577},"max":{"1min":110.98399999999999,"5min":110.98399999999999,"15min":188.26599999999999},"last":14.859999999999999},{"interface":"front","average":{"1min":25.422999999999998,"5min":17.34,"15min":21.085000000000001},"min":{"1min":7.9279999999999999,"5min":7.8470000000000004,"15min":7.5300000000000002},"max":{"1min":144.084,"5min":144.084,"15min":172.57400000000001},"last":10.394}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":19.388999999999999,"5min":14.335000000000001,"15min":18.338999999999999},"min":{"1min":8.4640000000000004,"5min":7.4800000000000004,"15min":6.141},"max":{"1min":62.637,"5min":62.637,"15min":201.25399999999999},"last":11.613},{"interface":"front","average":{"1min":24.920999999999999,"5min":17.582999999999998,"15min":21.760000000000002},"min":{"1min":10.914999999999999,"5min":8.9529999999999994,"15min":8.423},"max":{"1min":92.804000000000002,"5min":92.804000000000002,"15min":200.60599999999999},"last":16.273}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.809999999999999,"5min":18.367999999999999,"15min":21.443000000000001},"min":{"1min":10.18,"5min":8.4909999999999997,"15min":7.899},"max":{"1min":76.634,"5min":76.634,"15min":207.72200000000001},"last":18.260000000000002},{"interface":"front","average":{"1min":22.077000000000002,"5min":17.238,"15min":22.446999999999999},"min":{"1min":9.6129999999999995,"5min":9.3019999999999996,"15min":9.3019999999999996},"max":{"1min":69.144000000000005,"5min":69.144000000000005,"15min":193.43899999999999},"last":12.888}]}]},{"osd":3,"up_from":16,"seq":68719477129,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:20 2023","interfaces":[{"interface":"back","average":{"1min":25.492999999999999,"5min":15.124000000000001,"15min":16.145},"min":{"1min":4.9790000000000001,"5min":4.4720000000000004,"15min":4.1600000000000001},"max":{"1min":230.328,"5min":230.328,"15min":266.75400000000002},"last":9.0389999999999997},{"interface":"front","average":{"1min":20.965,"5min":14.555999999999999,"15min":17.283000000000001},"min":{"1min":5.4960000000000004,"5min":5.4100000000000001,"15min":4.6619999999999999},"max":{"1min":106.741,"5min":110.068,"15min":228.12899999999999},"last":12.244}]},{"osd":1,"last update":"Mon Dec 11 20:48:39 2023","interfaces":[{"interface":"back","average":{"1min":42.024000000000001,"5min":17.748999999999999,"15min":16.431000000000001},"min":{"1min":7.1529999999999996,"5min":6.7489999999999997,"15min":5.6180000000000003},"max":{"1min":170.31200000000001,"5min":170.31200000000001,"15min":179.11000000000001},"last":15.936999999999999},{"interface":"front","average":{"1min":50.826000000000001,"5min":19.158999999999999,"15min":18.561},"min":{"1min":8.9849999999999994,"5min":7.0529999999999999,"15min":6.508},"max":{"1min":215.578,"5min":215.578,"15min":215.578},"last":9.6300000000000008}]},{"osd":2,"last update":"Mon Dec 11 20:49:20 2023","interfaces":[{"interface":"back","average":{"1min":32.378,"5min":19.553000000000001,"15min":20.100999999999999},"min":{"1min":9.423,"5min":8.3559999999999999,"15min":7.1150000000000002},"max":{"1min":232.36199999999999,"5min":232.36199999999999,"15min":232.36199999999999},"last":10.192},{"interface":"front","average":{"1min":41.271999999999998,"5min":23.105,"15min":23.425000000000001},"min":{"1min":10.132,"5min":10.132,"15min":9.5990000000000002},"max":{"1min":239.23599999999999,"5min":239.23599999999999,"15min":239.23599999999999},"last":13.029999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:49:14 2023","interfaces":[{"interface":"back","average":{"1min":49.526000000000003,"5min":24.190999999999999,"15min":22.186},"min":{"1min":11.305,"5min":10.401999999999999,"15min":9.6539999999999999},"max":{"1min":258.495,"5min":258.495,"15min":258.495},"last":13.646000000000001},{"interface":"front","average":{"1min":27.492000000000001,"5min":16.739000000000001,"15min":16.282},"min":{"1min":8.2539999999999996,"5min":7.0510000000000002,"15min":7.0510000000000002},"max":{"1min":114.783,"5min":114.783,"15min":173.66800000000001},"last":8.1969999999999992}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":40.828000000000003,"5min":19.824000000000002,"15min":23.663},"min":{"1min":10.592000000000001,"5min":7.5250000000000004,"15min":7.5250000000000004},"max":{"1min":119.569,"5min":119.569,"15min":246.55000000000001},"last":10.864000000000001},{"interface":"front","average":{"1min":49.973999999999997,"5min":16.795999999999999,"15min":17.120999999999999},"min":{"1min":6.2610000000000001,"5min":5.4429999999999996,"15min":5.4429999999999996},"max":{"1min":203.21799999999999,"5min":203.21799999999999,"15min":205.15799999999999},"last":6.5460000000000003}]},{"osd":6,"last update":"Mon Dec 11 20:49:14 2023","interfaces":[{"interface":"back","average":{"1min":59.798999999999999,"5min":27.696999999999999,"15min":27.488},"min":{"1min":12.551,"5min":10.831,"15min":9.9320000000000004},"max":{"1min":285.733,"5min":285.733,"15min":285.733},"last":15.362},{"interface":"front","average":{"1min":39.789999999999999,"5min":22.498999999999999,"15min":24.436},"min":{"1min":12.676,"5min":9.8339999999999996,"15min":8.391},"max":{"1min":171.57499999999999,"5min":171.57499999999999,"15min":208.00299999999999},"last":14.731999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":60.445,"5min":26.123000000000001,"15min":21.013999999999999},"min":{"1min":12.372,"5min":12.295999999999999,"15min":6.4690000000000003},"max":{"1min":260.45999999999998,"5min":260.45999999999998,"15min":260.45999999999998},"last":17.768999999999998},{"interface":"front","average":{"1min":44.287999999999997,"5min":22.015999999999998,"15min":25.881},"min":{"1min":12.273999999999999,"5min":10.863,"15min":10.863},"max":{"1min":172.559,"5min":172.559,"15min":264.99299999999999},"last":16.539000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477128,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":20.559000000000001,"5min":12.036,"15min":15.512},"min":{"1min":5.0780000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"max":{"1min":89.881,"5min":89.881,"15min":200.447},"last":6.1849999999999996},{"interface":"front","average":{"1min":19.225000000000001,"5min":11.49,"15min":16.253},"min":{"1min":5.6619999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":92.114000000000004,"5min":92.114000000000004,"15min":198.22900000000001},"last":10.288}]},{"osd":1,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":21.713000000000001,"5min":13.699,"15min":16.245999999999999},"min":{"1min":7.0590000000000002,"5min":5.7869999999999999,"15min":5.7869999999999999},"max":{"1min":89.183999999999997,"5min":89.183999999999997,"15min":143.56},"last":13.412000000000001},{"interface":"front","average":{"1min":27.739999999999998,"5min":15.618,"15min":17.497},"min":{"1min":6.1879999999999997,"5min":6.1879999999999997,"15min":6.1879999999999997},"max":{"1min":175.16200000000001,"5min":175.16200000000001,"15min":175.16200000000001},"last":9.9329999999999998}]},{"osd":3,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":19.030999999999999,"5min":11.109999999999999,"15min":13.959},"min":{"1min":6.0780000000000003,"5min":5.7960000000000003,"15min":5.3550000000000004},"max":{"1min":99.146000000000001,"5min":99.146000000000001,"15min":171.71700000000001},"last":12.106},{"interface":"front","average":{"1min":21.315999999999999,"5min":15.250999999999999,"15min":17.931000000000001},"min":{"1min":8.6479999999999997,"5min":8.0220000000000002,"15min":5.9749999999999996},"max":{"1min":82.924999999999997,"5min":82.924999999999997,"15min":182.63399999999999},"last":14.786}]},{"osd":4,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":32.999000000000002,"5min":17.382999999999999,"15min":19.341999999999999},"min":{"1min":6.694,"5min":6.694,"15min":6.694},"max":{"1min":192.851,"5min":192.851,"15min":192.851},"last":11.331},{"interface":"front","average":{"1min":40.692999999999998,"5min":21.776,"15min":24.178999999999998},"min":{"1min":11.397,"5min":10.923999999999999,"15min":10.285},"max":{"1min":259.05900000000003,"5min":259.05900000000003,"15min":259.05900000000003},"last":15.555}]},{"osd":5,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":49.709000000000003,"5min":22.446999999999999,"15min":21.960999999999999},"min":{"1min":7.8159999999999998,"5min":7.8159999999999998,"15min":7.8159999999999998},"max":{"1min":364.11200000000002,"5min":364.11200000000002,"15min":364.11200000000002},"last":14.119999999999999},{"interface":"front","average":{"1min":39.561999999999998,"5min":20.593,"15min":24.591999999999999},"min":{"1min":10.625999999999999,"5min":9.1349999999999998,"15min":8.5020000000000007},"max":{"1min":296.74000000000001,"5min":296.74000000000001,"15min":296.74000000000001},"last":18.591999999999999}]},{"osd":6,"last update":"Mon Dec 11 20:49:22 2023","interfaces":[{"interface":"back","average":{"1min":18.748000000000001,"5min":15.739000000000001,"15min":17.251999999999999},"min":{"1min":8.0180000000000007,"5min":6.6799999999999997,"15min":6.3259999999999996},"max":{"1min":91.569999999999993,"5min":251.19300000000001,"15min":251.19300000000001},"last":10.669},{"interface":"front","average":{"1min":26.641999999999999,"5min":18.463999999999999,"15min":21.571999999999999},"min":{"1min":11.819000000000001,"5min":9.4960000000000004,"15min":9.4469999999999992},"max":{"1min":103.833,"5min":103.833,"15min":181.96299999999999},"last":19.844999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":38.201999999999998,"5min":22.239999999999998,"15min":24.962},"min":{"1min":13.659000000000001,"5min":10.379,"15min":10.379},"max":{"1min":156.934,"5min":156.934,"15min":197.27199999999999},"last":17.771000000000001},{"interface":"front","average":{"1min":31.547999999999998,"5min":17.236000000000001,"15min":19.617000000000001},"min":{"1min":9.9179999999999993,"5min":8.0199999999999996,"15min":7.085},"max":{"1min":173.654,"5min":173.654,"15min":233.57400000000001},"last":12.832000000000001}]}]},{"osd":1,"up_from":16,"seq":68719477129,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69716,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302124,"statfs":{"total":96636764160,"available":96565374976,"internally_reserved":0,"allocated":2502656,"data_stored":954021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":7.8470000000000004,"15min":13.646000000000001},"min":{"1min":4.6870000000000003,"5min":4.0949999999999998,"15min":3.903},"max":{"1min":72.888000000000005,"5min":72.888000000000005,"15min":283.91899999999998},"last":14.259},{"interface":"front","average":{"1min":17.266999999999999,"5min":9.9909999999999997,"15min":14.855},"min":{"1min":5.8250000000000002,"5min":4.9340000000000002,"15min":4.6879999999999997},"max":{"1min":99.680000000000007,"5min":99.680000000000007,"15min":391.19999999999999},"last":6.5519999999999996}]},{"osd":2,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":18.952999999999999,"5min":13.276,"15min":19.867999999999999},"min":{"1min":7.7510000000000003,"5min":6.0300000000000002,"15min":6.0300000000000002},"max":{"1min":84.167000000000002,"5min":84.167000000000002,"15min":389.66800000000001},"last":10.736000000000001},{"interface":"front","average":{"1min":19.573,"5min":14.933999999999999,"15min":20.850000000000001},"min":{"1min":9.2989999999999995,"5min":9.2590000000000003,"15min":6.7670000000000003},"max":{"1min":100.836,"5min":100.836,"15min":376.82100000000003},"last":11.492000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":20.777999999999999,"5min":12.682,"15min":24.100999999999999},"min":{"1min":6.9080000000000004,"5min":5.7610000000000001,"15min":5.7610000000000001},"max":{"1min":101.74299999999999,"5min":101.74299999999999,"15min":421.89400000000001},"last":7.734},{"interface":"front","average":{"1min":29.260000000000002,"5min":16.670000000000002,"15min":20.762},"min":{"1min":10.404,"5min":5.7220000000000004,"15min":5.7220000000000004},"max":{"1min":175.047,"5min":175.047,"15min":293.06099999999998},"last":13.664}]},{"osd":4,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":12.462999999999999,"15min":18.878},"min":{"1min":6.6740000000000004,"5min":6.1630000000000003,"15min":5.952},"max":{"1min":120.863,"5min":120.863,"15min":342.28199999999998},"last":8.9969999999999999},{"interface":"front","average":{"1min":33.405000000000001,"5min":18.756,"15min":23.010000000000002},"min":{"1min":12.538,"5min":10.499000000000001,"15min":6.7039999999999997},"max":{"1min":187.15100000000001,"5min":187.15100000000001,"15min":289.68799999999999},"last":15.423}]},{"osd":5,"last update":"Mon Dec 11 20:48:42 2023","interfaces":[{"interface":"back","average":{"1min":25.463999999999999,"5min":17.777000000000001,"15min":22.192},"min":{"1min":11.183,"5min":11.183,"15min":7.4859999999999998},"max":{"1min":183.84700000000001,"5min":183.84700000000001,"15min":556.32500000000005},"last":12.146000000000001},{"interface":"front","average":{"1min":27.771000000000001,"5min":19.635999999999999,"15min":24.861000000000001},"min":{"1min":12.422000000000001,"5min":12.422000000000001,"15min":9.5739999999999998},"max":{"1min":186.06700000000001,"5min":186.06700000000001,"15min":309.16399999999999},"last":16.350000000000001}]},{"osd":6,"last update":"Mon Dec 11 20:48:43 2023","interfaces":[{"interface":"back","average":{"1min":23.209,"5min":13.757,"15min":20.975000000000001},"min":{"1min":8.6910000000000007,"5min":7.5270000000000001,"15min":6.7590000000000003},"max":{"1min":169.429,"5min":169.429,"15min":372.90899999999999},"last":13.058},{"interface":"front","average":{"1min":29.114999999999998,"5min":19.785,"15min":27.733000000000001},"min":{"1min":12.670999999999999,"5min":12.670999999999999,"15min":11.641999999999999},"max":{"1min":197.87299999999999,"5min":197.87299999999999,"15min":297.71499999999997},"last":17.140999999999998}]},{"osd":7,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":20.015000000000001,"5min":13.327,"15min":22.792999999999999},"min":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"max":{"1min":109.782,"5min":109.782,"15min":426.10700000000003},"last":9.5359999999999996},{"interface":"front","average":{"1min":33.633000000000003,"5min":21.547000000000001,"15min":29.686},"min":{"1min":13.068,"5min":13.068,"15min":11.789},"max":{"1min":189.21199999999999,"5min":189.21199999999999,"15min":331.24799999999999},"last":17.922999999999998}]}]},{"osd":0,"up_from":16,"seq":68719477131,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":11.036,"15min":16.446000000000002},"min":{"1min":4.609,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":98.932000000000002,"5min":98.932000000000002,"15min":192.989},"last":8.5050000000000008},{"interface":"front","average":{"1min":20.565000000000001,"5min":11.683999999999999,"15min":17.998999999999999},"min":{"1min":6.1719999999999997,"5min":5.2220000000000004,"15min":4.7300000000000004},"max":{"1min":101.66,"5min":101.66,"15min":288.91800000000001},"last":15.244}]},{"osd":2,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":22.863,"5min":13.515000000000001,"15min":17.582999999999998},"min":{"1min":7.694,"5min":7.0620000000000003,"15min":5.2190000000000003},"max":{"1min":96.489999999999995,"5min":96.489999999999995,"15min":294.733},"last":12.465},{"interface":"front","average":{"1min":29.260999999999999,"5min":17.803999999999998,"15min":23.199999999999999},"min":{"1min":11.227,"5min":9.9109999999999996,"15min":7.8630000000000004},"max":{"1min":126.831,"5min":126.831,"15min":306.08100000000002},"last":15.866}]},{"osd":3,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.068000000000001,"5min":13.037000000000001,"15min":15.56},"min":{"1min":6.1769999999999996,"5min":5.9829999999999997,"15min":5.5979999999999999},"max":{"1min":160.999,"5min":160.999,"15min":166.49100000000001},"last":9.6809999999999992},{"interface":"front","average":{"1min":29.303000000000001,"5min":17.632999999999999,"15min":20.312999999999999},"min":{"1min":10.973000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"max":{"1min":177.35400000000001,"5min":177.35400000000001,"15min":177.35400000000001},"last":18.536000000000001}]},{"osd":4,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.286000000000001,"5min":16.172999999999998,"15min":19.902999999999999},"min":{"1min":6.5970000000000004,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":169.92599999999999,"5min":169.92599999999999,"15min":169.92599999999999},"last":11.763999999999999},{"interface":"front","average":{"1min":31.260999999999999,"5min":19.547000000000001,"15min":20.800000000000001},"min":{"1min":10.371,"5min":10.371,"15min":6.6859999999999999},"max":{"1min":149.65600000000001,"5min":149.65600000000001,"15min":153.42500000000001},"last":19.16}]},{"osd":5,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.992999999999999,"5min":13.356,"15min":15.753},"min":{"1min":7.0940000000000003,"5min":6.5999999999999996,"15min":6.234},"max":{"1min":110.476,"5min":110.476,"15min":210.374},"last":10.384},{"interface":"front","average":{"1min":30.498999999999999,"5min":16.635999999999999,"15min":20.227},"min":{"1min":8.7940000000000005,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":110.836,"5min":110.836,"15min":267.03199999999998},"last":14.234}]},{"osd":6,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":24.010999999999999,"5min":14.885,"15min":18.350999999999999},"min":{"1min":9.0850000000000009,"5min":8.0050000000000008,"15min":6.4790000000000001},"max":{"1min":111.423,"5min":111.423,"15min":161.09899999999999},"last":13.311},{"interface":"front","average":{"1min":33.442999999999998,"5min":19.559000000000001,"15min":23.120000000000001},"min":{"1min":11.138999999999999,"5min":10.305999999999999,"15min":9.4440000000000008},"max":{"1min":186.089,"5min":186.089,"15min":246.738},"last":20.463999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.001000000000001,"5min":19.183,"15min":21.738},"min":{"1min":9.4049999999999994,"5min":9.4049999999999994,"15min":8.8789999999999996},"max":{"1min":187.00299999999999,"5min":187.00299999999999,"15min":187.00299999999999},"last":17.562999999999999},{"interface":"front","average":{"1min":29.199999999999999,"5min":19.706,"15min":23.911999999999999},"min":{"1min":12.224,"5min":11.782999999999999,"15min":7.7640000000000002},"max":{"1min":127.563,"5min":127.563,"15min":264.45100000000002},"last":21.648}]}]},{"osd":4,"up_from":14,"seq":60129542535,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65168,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306672,"statfs":{"total":96636764160,"available":96570032128,"internally_reserved":0,"allocated":2105344,"data_stored":556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":13.959,"5min":10.147,"15min":15.048999999999999},"min":{"1min":4.5049999999999999,"5min":4.4409999999999998,"15min":4.1879999999999997},"max":{"1min":40.877000000000002,"5min":40.877000000000002,"15min":248.19900000000001},"last":12.737},{"interface":"front","average":{"1min":17.295999999999999,"5min":10.696999999999999,"15min":13.792},"min":{"1min":5.1589999999999998,"5min":5.1059999999999999,"15min":4.1449999999999996},"max":{"1min":77.028000000000006,"5min":77.028000000000006,"15min":169.541},"last":6.7999999999999998}]},{"osd":1,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":23.628,"5min":14.050000000000001,"15min":18.832000000000001},"min":{"1min":7.1269999999999998,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":116.298,"5min":116.298,"15min":310.53500000000003},"last":17.129999999999999},{"interface":"front","average":{"1min":21.18,"5min":13.920999999999999,"15min":18.367999999999999},"min":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":4.7380000000000004},"max":{"1min":109.753,"5min":109.753,"15min":269.65100000000001},"last":7.7009999999999996}]},{"osd":2,"last update":"Mon Dec 11 20:49:17 2023","interfaces":[{"interface":"back","average":{"1min":15.798999999999999,"5min":14.196999999999999,"15min":18.044},"min":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":6.1429999999999998},"max":{"1min":50.573,"5min":50.573,"15min":255.31700000000001},"last":15.657},{"interface":"front","average":{"1min":13.231999999999999,"5min":11.869,"15min":17.018000000000001},"min":{"1min":6.9139999999999997,"5min":6.6310000000000002,"15min":5.4199999999999999},"max":{"1min":40.917999999999999,"5min":40.917999999999999,"15min":291.67899999999997},"last":8.2750000000000004}]},{"osd":3,"last update":"Mon Dec 11 20:49:10 2023","interfaces":[{"interface":"back","average":{"1min":21.579000000000001,"5min":14.327999999999999,"15min":18.846},"min":{"1min":8.2970000000000006,"5min":7.6790000000000003,"15min":7.2460000000000004},"max":{"1min":132.065,"5min":132.065,"15min":260.55099999999999},"last":12.01},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.18,"15min":21.841000000000001},"min":{"1min":7.0860000000000003,"5min":7.0860000000000003,"15min":7.0030000000000001},"max":{"1min":50.997999999999998,"5min":50.997999999999998,"15min":320.53899999999999},"last":16.401}]},{"osd":5,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":26.222000000000001,"5min":18.422999999999998,"15min":21.838999999999999},"min":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":7.7359999999999998},"max":{"1min":132.37799999999999,"5min":132.37799999999999,"15min":298.69499999999999},"last":17.875},{"interface":"front","average":{"1min":22.783999999999999,"5min":15.782,"15min":21.024999999999999},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":81.689999999999998,"5min":81.689999999999998,"15min":309.88200000000001},"last":9.298}]},{"osd":6,"last update":"Mon Dec 11 20:49:10 2023","interfaces":[{"interface":"back","average":{"1min":25.664999999999999,"5min":17.158000000000001,"15min":19.324999999999999},"min":{"1min":10.003,"5min":8.6419999999999995,"15min":5.4459999999999997},"max":{"1min":128.589,"5min":128.589,"15min":290.57600000000002},"last":13.349},{"interface":"front","average":{"1min":24.858000000000001,"5min":15.848000000000001,"15min":19.082999999999998},"min":{"1min":6.5469999999999997,"5min":6.5469999999999997,"15min":6.5389999999999997},"max":{"1min":107.01600000000001,"5min":107.01600000000001,"15min":285.52499999999998},"last":11.26}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":21.823,"5min":17.431000000000001,"15min":21.483000000000001},"min":{"1min":10.587,"5min":10.176,"15min":7.2599999999999998},"max":{"1min":69.731999999999999,"5min":69.731999999999999,"15min":297.63299999999998},"last":14.778},{"interface":"front","average":{"1min":23.029,"5min":16.969999999999999,"15min":23.030000000000001},"min":{"1min":10.496,"5min":7.3639999999999999,"15min":7.3639999999999999},"max":{"1min":98.728999999999999,"5min":98.728999999999999,"15min":364.18799999999999},"last":10.627000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575242,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:20 2023","interfaces":[{"interface":"back","average":{"1min":12.233000000000001,"5min":10.454000000000001,"15min":15.913},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":3.8969999999999998},"max":{"1min":41.088000000000001,"5min":41.088000000000001,"15min":280.62200000000001},"last":11.974},{"interface":"front","average":{"1min":12.196,"5min":9.984,"15min":16.021000000000001},"min":{"1min":5.516,"5min":5.516,"15min":4.9800000000000004},"max":{"1min":49.896000000000001,"5min":49.896000000000001,"15min":294.44299999999998},"last":5.2350000000000003}]},{"osd":1,"last update":"Mon Dec 11 20:49:09 2023","interfaces":[{"interface":"back","average":{"1min":12.32,"5min":11.670999999999999,"15min":16.097999999999999},"min":{"1min":6.5209999999999999,"5min":5.5519999999999996,"15min":5.4489999999999998},"max":{"1min":21.082000000000001,"5min":21.082000000000001,"15min":217.40100000000001},"last":11.635},{"interface":"front","average":{"1min":13.526999999999999,"5min":12.893000000000001,"15min":17.603000000000002},"min":{"1min":9.3949999999999996,"5min":6.3730000000000002,"15min":6.3730000000000002},"max":{"1min":22.927,"5min":22.927,"15min":247.75299999999999},"last":13.579000000000001}]},{"osd":2,"last update":"Mon Dec 11 20:49:09 2023","interfaces":[{"interface":"back","average":{"1min":20.245999999999999,"5min":14.756,"15min":18.093},"min":{"1min":9.5399999999999991,"5min":7.7839999999999998,"15min":7.1920000000000002},"max":{"1min":143.51599999999999,"5min":143.51599999999999,"15min":168.923},"last":15.257},{"interface":"front","average":{"1min":37.488,"5min":20.488,"15min":24.061},"min":{"1min":11.813000000000001,"5min":11.714,"15min":11.016},"max":{"1min":297.77999999999997,"5min":297.77999999999997,"15min":297.77999999999997},"last":16.751999999999999}]},{"osd":3,"last update":"Mon Dec 11 20:49:09 2023","interfaces":[{"interface":"back","average":{"1min":13.353,"5min":9.3569999999999993,"15min":12.16},"min":{"1min":6.4029999999999996,"5min":5.1879999999999997,"15min":5.1879999999999997},"max":{"1min":86.045000000000002,"5min":86.045000000000002,"15min":199.89500000000001},"last":7.6820000000000004},{"interface":"front","average":{"1min":23.440999999999999,"5min":17.196000000000002,"15min":20.536999999999999},"min":{"1min":11.305,"5min":9.6389999999999993,"15min":9.2240000000000002},"max":{"1min":143.214,"5min":143.214,"15min":210.63200000000001},"last":16.007000000000001}]},{"osd":4,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":27.109999999999999,"5min":15.597,"15min":16.378},"min":{"1min":9.6489999999999991,"5min":6.516,"15min":6.1900000000000004},"max":{"1min":166.67099999999999,"5min":166.67099999999999,"15min":201.459},"last":10.497999999999999},{"interface":"front","average":{"1min":22.686,"5min":13.342000000000001,"15min":17.608000000000001},"min":{"1min":7.4580000000000002,"5min":6.2510000000000003,"15min":6.2510000000000003},"max":{"1min":124.88,"5min":124.88,"15min":203.36600000000001},"last":9.7970000000000006}]},{"osd":5,"last update":"Mon Dec 11 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":39.771000000000001,"5min":21.721,"15min":26.289000000000001},"min":{"1min":12.252000000000001,"5min":10.476000000000001,"15min":9.7810000000000006},"max":{"1min":206.904,"5min":206.904,"15min":268.98000000000002},"last":17.614999999999998},{"interface":"front","average":{"1min":28.629000000000001,"5min":16.504999999999999,"15min":19.289999999999999},"min":{"1min":8.9410000000000007,"5min":8.2639999999999993,"15min":7.6360000000000001},"max":{"1min":127.8,"5min":127.8,"15min":283.61900000000003},"last":12.929}]},{"osd":6,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":22.581,"5min":16.634,"15min":21.099},"min":{"1min":12.592000000000001,"5min":8.8339999999999996,"15min":6.6479999999999997},"max":{"1min":115.38800000000001,"5min":115.38800000000001,"15min":282.71199999999999},"last":14.163},{"interface":"front","average":{"1min":25.696999999999999,"5min":19.786000000000001,"15min":25.102},"min":{"1min":15.474,"5min":10.999000000000001,"15min":9.7569999999999997},"max":{"1min":126.97199999999999,"5min":126.97199999999999,"15min":260.13400000000001},"last":18.364999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575240,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69716,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302124,"statfs":{"total":96636764160,"available":96565374976,"internally_reserved":0,"allocated":2502656,"data_stored":954021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":13.065,"5min":10.132999999999999,"15min":13.657999999999999},"min":{"1min":4.4960000000000004,"5min":4.4509999999999996,"15min":4.3849999999999998},"max":{"1min":52.319000000000003,"5min":52.319000000000003,"15min":121.80500000000001},"last":7.9950000000000001},{"interface":"front","average":{"1min":13.058999999999999,"5min":9.7390000000000008,"15min":16.152999999999999},"min":{"1min":6.3780000000000001,"5min":4.5369999999999999,"15min":4.4349999999999996},"max":{"1min":49.723999999999997,"5min":49.723999999999997,"15min":307.66000000000003},"last":8.5909999999999993}]},{"osd":1,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":12.073,"5min":9.8230000000000004,"15min":13.728},"min":{"1min":6.1070000000000002,"5min":5.3449999999999998,"15min":4.5490000000000004},"max":{"1min":57.738999999999997,"5min":57.738999999999997,"15min":131.70500000000001},"last":7.5620000000000003},{"interface":"front","average":{"1min":14.811,"5min":11.605,"15min":16.902000000000001},"min":{"1min":7.4320000000000004,"5min":6.0609999999999999,"15min":5.7169999999999996},"max":{"1min":60.664999999999999,"5min":60.664999999999999,"15min":202.07300000000001},"last":9.923}]},{"osd":2,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.559000000000001,"5min":15.26,"15min":18.596},"min":{"1min":8.6720000000000006,"5min":7.7460000000000004,"15min":6.9470000000000001},"max":{"1min":109.023,"5min":109.023,"15min":203.67699999999999},"last":14.868},{"interface":"front","average":{"1min":23.201000000000001,"5min":15.288,"15min":19.117000000000001},"min":{"1min":9.0129999999999999,"5min":7.8760000000000003,"15min":6.8719999999999999},"max":{"1min":93.819999999999993,"5min":93.819999999999993,"15min":211.67500000000001},"last":13.156000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":15.050000000000001,"15min":19.510000000000002},"min":{"1min":8.3170000000000002,"5min":8.0700000000000003,"15min":7.548},"max":{"1min":58.795000000000002,"5min":58.795000000000002,"15min":167.095},"last":15.467000000000001},{"interface":"front","average":{"1min":19.649000000000001,"5min":15.371,"15min":19.989000000000001},"min":{"1min":10.052,"5min":8.1690000000000005,"15min":7.6769999999999996},"max":{"1min":68.209999999999994,"5min":68.209999999999994,"15min":207.58799999999999},"last":18.699999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.547000000000001,"5min":16.661000000000001,"15min":19.132000000000001},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":6.8360000000000003},"max":{"1min":69.195999999999998,"5min":69.195999999999998,"15min":132.64500000000001},"last":16.41},{"interface":"front","average":{"1min":33.280999999999999,"5min":20.861000000000001,"15min":23.035},"min":{"1min":14.875999999999999,"5min":9.9390000000000001,"15min":8.2590000000000003},"max":{"1min":199.828,"5min":199.828,"15min":199.828},"last":19.645}]},{"osd":6,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":23.556999999999999,"5min":12.887,"15min":16.77},"min":{"1min":6.3620000000000001,"5min":6.234,"15min":5.8789999999999996},"max":{"1min":199.03800000000001,"5min":199.03800000000001,"15min":199.03800000000001},"last":11.188000000000001},{"interface":"front","average":{"1min":26.257999999999999,"5min":19.350999999999999,"15min":19.757000000000001},"min":{"1min":11.548999999999999,"5min":11.548999999999999,"15min":6.1479999999999997},"max":{"1min":113.911,"5min":113.911,"15min":268.072},"last":17.788}]},{"osd":7,"last update":"Mon Dec 11 20:48:44 2023","interfaces":[{"interface":"back","average":{"1min":22.385000000000002,"5min":19.629999999999999,"15min":22.059999999999999},"min":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":8.3239999999999998},"max":{"1min":63.686999999999998,"5min":63.686999999999998,"15min":198.304},"last":20.492000000000001},{"interface":"front","average":{"1min":19.053000000000001,"5min":15.726000000000001,"15min":20.466000000000001},"min":{"1min":10.156000000000001,"5min":8.0630000000000006,"15min":8.0630000000000006},"max":{"1min":56.078000000000003,"5min":56.078000000000003,"15min":181.84299999999999},"last":17.016999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-11T20:49:33.640 INFO:tasks.ceph:pgid 8.e last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:33.640 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:33.641 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:33.641 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2023-12-11T20:17:56.659735+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=17, tm_sec=56, tm_wday=0, tm_yday=345, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=11, tm_hour=20, tm_min=49, tm_sec=5, tm_wday=0, tm_yday=345, tm_isdst=0) 2023-12-11T20:49:33.641 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-11T20:49:34.768 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:47.876 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:34.770 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:47.878 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:34.786 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:47.894 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:34.816 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:47.885 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:34.817 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:47.925 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:34.818 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:47.887 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:34.819 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:47.927 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:34.834 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:47.942 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:34.835 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:47.903 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.391 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:48.459 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.392 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:48.460 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.401 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:33:48.469 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.423 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:48.531 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.424 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:48.532 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.427 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:48.535 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.428 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:48.536 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.432 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:33:48.540 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:35.436 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:33:48.544 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:53.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-11T20:49:53.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.744+0000 7f8499ac9640 10 monclient: get_monmap_and_config 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.744+0000 7f8499ac9640 10 monclient: build_initial_monmap 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.744+0000 7f8499ac9640 10 monclient: monmap: 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:53.745650+0000 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:53.745650+0000 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:53.744 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.744+0000 7f8499ac9640 1 Processor -- start 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.744+0000 7f8499ac9640 1 -- start start 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.744+0000 7f8499ac9640 10 monclient: init 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84940a8a20 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 1 --2- >> v2:172.21.15.133:3300/0 conn(0x7f84940a8e00 0x7f849413f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f84940a8e00 addr v2:172.21.15.133:3300/0 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84941423b0 0x7f8494142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f84941423b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84940b21f0 con 0x7f84940a8a20 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:53.747 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 1 -- --> v2:172.21.15.133:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84940a9cc0 con 0x7f84940a8e00 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84940a9ed0 con 0x7f84941423b0 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.745+0000 7f8499ac9640 10 monclient(hunting): authenticate will time out at 3480.075705s 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f8493fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84941423b0 0x7f8494142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f84937fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f84940a8a20 0x7f849413eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41520/0 (socket says 172.21.15.86:41520) 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f84937fe640 1 -- 172.21.15.86:0/824612176 learned_addr learned my addr 172.21.15.86:0/824612176 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f848affd640 1 --2- 172.21.15.86:0/824612176 >> v2:172.21.15.133:3300/0 conn(0x7f84940a8e00 0x7f849413f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f8493fff640 10 monclient(hunting): get_auth_request con 0x7f84941423b0 auth_method 0 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f8493fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f8493fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f8493fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f848affd640 10 monclient(hunting): get_auth_request con 0x7f84940a8e00 auth_method 0 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f848affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f848affd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.746+0000 7f848affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.747+0000 7f8493fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.747+0000 7f8493fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.747+0000 7f8493fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.747+0000 7f848affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.747+0000 7f848affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.747+0000 7f848affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:53.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8492ffd640 1 -- 172.21.15.86:0/824612176 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807082470 0 0) 0x7f84940b21f0 con 0x7f84940a8a20 2023-12-11T20:49:53.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8492ffd640 10 monclient(hunting): _negotiate 2023-12-11T20:49:53.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8492ffd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:53.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8492ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:53.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8492ffd640 10 monclient(hunting): my global_id is 16968 2023-12-11T20:49:53.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8492ffd640 1 -- 172.21.15.86:0/824612176 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8480003520 con 0x7f84940a8a20 2023-12-11T20:49:53.750 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 10 monclient(hunting): handle_auth_done global_id 19085 payload 274 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 1 -- 172.21.15.86:0/824612176 >> v2:172.21.15.133:3300/0 conn(0x7f84940a8e00 msgr2=0x7f849413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 1 --2- 172.21.15.86:0/824612176 >> v2:172.21.15.133:3300/0 conn(0x7f84940a8e00 0x7f849413f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 1 -- 172.21.15.86:0/824612176 >> v1:172.21.15.86:6789/0 conn(0x7f84940a8a20 legacy=0x7f849413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 10 monclient: _finish_hunting 0 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 1 monclient: found mon.noname-b 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.748+0000 7f8493fff640 1 -- 172.21.15.86:0/824612176 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84940aac70 con 0x7f84941423b0 2023-12-11T20:49:53.751 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.751+0000 7f8493fff640 1 --2- 172.21.15.86:0/824612176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84941423b0 0x7f8494142840 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f848c009d90 tx=0x7f848c02bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed336425dd1e2a6e server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:53.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8492ffd640 1 -- 172.21.15.86:0/824612176 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c02cb30 con 0x7f84941423b0 2023-12-11T20:49:53.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:53.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8492ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:53.752 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8492ffd640 10 monclient: dump: 2023-12-11T20:49:53.752 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:53.752 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:53.752 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8492ffd640 10 monclient: _finish_auth 0 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8492ffd640 10 monclient: _check_auth_tickets 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8499ac9640 5 monclient: authenticate success, global_id 19085 2023-12-11T20:49:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.752+0000 7f8499ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:53.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8492ffd640 1 -- 172.21.15.86:0/824612176 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c02cc90 con 0x7f84941423b0 2023-12-11T20:49:53.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8492ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:53.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8492ffd640 1 -- 172.21.15.86:0/824612176 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c034440 con 0x7f84941423b0 2023-12-11T20:49:53.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:53.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 4 set_mon_vals no callback set 2023-12-11T20:49:53.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:53.755 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8492ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8492ffd640 10 monclient: dump: 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 10 monclient: shutdown 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 1 -- 172.21.15.86:0/824612176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84941423b0 msgr2=0x7f8494142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:53.756 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 1 --2- 172.21.15.86:0/824612176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84941423b0 0x7f8494142840 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f848c009d90 tx=0x7f848c02bd80 comp rx=0 tx=0).stop 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 1 -- 172.21.15.86:0/824612176 shutdown_connections 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 1 --2- 172.21.15.86:0/824612176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84941423b0 0x7f8494142840 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 1 --2- 172.21.15.86:0/824612176 >> v2:172.21.15.133:3300/0 conn(0x7f84940a8e00 0x7f849413f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 1 -- 172.21.15.86:0/824612176 >> 172.21.15.86:0/824612176 conn(0x7f84940b7870 msgr2=0x7f84940b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 1 -- 172.21.15.86:0/824612176 shutdown_connections 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.754+0000 7f8499ac9640 1 -- 172.21.15.86:0/824612176 wait complete. 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.755+0000 7f8499ac9640 10 monclient: build_initial_monmap 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.755+0000 7f8499ac9640 10 monclient: monmap: 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:53.757 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.755+0000 7f8499ac9640 1 librados: starting msgr at 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.755+0000 7f8499ac9640 1 librados: starting objecter 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.755+0000 7f8499ac9640 1 Processor -- start 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 -- start start 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 librados: setting wanted keys 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 librados: calling monclient init 2023-12-11T20:49:53.758 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient: init 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84940a8a20 0x7f84940ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f84940a8a20 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84940a8e00 0x7f8494135710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f84940a8e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84941423b0 0x7f84940b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient(hunting): picked mon.noname-a con 0x7f84941423b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84940abe10 con 0x7f84941423b0 2023-12-11T20:49:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84940acb90 con 0x7f84940a8a20 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84940ace70 con 0x7f84940a8e00 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.756+0000 7f8499ac9640 10 monclient(hunting): authenticate will time out at 3480.086788s 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84940a8e00 0x7f8494135710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f8493fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84941423b0 0x7f84940b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f84937fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84940a8a20 0x7f84940ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84940a8e00 0x7f8494135710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56468/0 (socket says 172.21.15.86:56468) 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 1 -- 172.21.15.86:0/1886399032 learned_addr learned my addr 172.21.15.86:0/1886399032 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 10 monclient(hunting): get_auth_request con 0x7f84940a8e00 auth_method 0 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f8493fff640 10 monclient(hunting): get_auth_request con 0x7f84941423b0 auth_method 0 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f8493fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f8493fff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f8493fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f84937fe640 10 monclient(hunting): get_auth_request con 0x7f84940a8a20 auth_method 0 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f84937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f84937fe640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f84937fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.757+0000 7f848affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.758+0000 7f8493fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:53.763 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.758+0000 7f8493fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.758+0000 7f8493fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.758+0000 7f84937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.758+0000 7f84937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.758+0000 7f84937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 10 monclient(hunting): handle_auth_done global_id 19088 payload 824 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 1 -- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84940a8a20 msgr2=0x7f84940ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84940a8a20 0x7f84940ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 1 -- 172.21.15.86:0/1886399032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84941423b0 msgr2=0x7f84940b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84941423b0 0x7f84940b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 10 monclient: _finish_hunting 0 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 1 monclient: found mon.noname-c 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:53.764 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2023-12-11T20:49:53.765 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.760+0000 7f848affd640 1 -- 172.21.15.86:0/1886399032 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c009a70 con 0x7f84940a8e00 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.763+0000 7f848affd640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84940a8e00 0x7f8494135710 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f847c00e9b0 tx=0x7f847c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.763+0000 7f8490ff9640 10 client.?.objecter ms_handle_connect 0x7f84940a8e00 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.763+0000 7f8490ff9640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8490ff9640 1 -- 172.21.15.86:0/1886399032 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c009670 con 0x7f84940a8e00 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8490ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8490ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8490ff9640 10 monclient: dump: 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:53.766 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8490ff9640 10 monclient: _finish_auth 0 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8490ff9640 10 monclient: _check_auth_tickets 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8490ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 5 monclient: authenticate success, global_id 19088 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 10 monclient: _renew_subs 2023-12-11T20:49:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84940ad150 con 0x7f84940a8e00 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 10 client.19088.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 10 monclient: _renew_subs 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84940abb20 con 0x7f84940a8e00 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.765+0000 7f8499ac9640 1 librados: init done 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.766+0000 7f8499ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.766+0000 7f8499ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.766+0000 7f8499ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2023-12-11T20:49:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.766+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849404e840 con 0x7f84940a8e00 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.767+0000 7f8490ff9640 1 -- 172.21.15.86:0/1886399032 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c004030 con 0x7f84940a8e00 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.767+0000 7f8490ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.767+0000 7f8490ff9640 1 -- 172.21.15.86:0/1886399032 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c0043a0 con 0x7f84940a8e00 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.767+0000 7f8490ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.767+0000 7f8490ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.767+0000 7f8490ff9640 10 monclient: dump: 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:53.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:53.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:53.770 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:53.770 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:53.770 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:53.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:53.767+0000 7f84927fc640 4 set_mon_vals no callback set 2023-12-11T20:49:54.008 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.008+0000 7f8490ff9640 1 -- 172.21.15.86:0/1886399032 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f847c005990 con 0x7f84940a8e00 2023-12-11T20:49:54.008 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.009+0000 7f8490ff9640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f847406c820 0x7f847406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:54.009 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.009+0000 7f84937fe640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f847406c820 0x7f847406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:54.009 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.009+0000 7f84937fe640 10 monclient: get_auth_request con 0x7f847406c820 auth_method 0 2023-12-11T20:49:54.009 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.010+0000 7f8490ff9640 10 client.19088.objecter ms_handle_connect 0x7f847406c820 2023-12-11T20:49:54.009 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.010+0000 7f84937fe640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f847406c820 0x7f847406ece0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8484005880 tx=0x7f8484009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:54.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.024+0000 7f8490ff9640 1 -- 172.21.15.86:0/1886399032 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f847c014e30 con 0x7f84940a8e00 2023-12-11T20:49:54.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.024+0000 7f8490ff9640 10 client.19088.objecter ms_dispatch 0x7f8494000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:54.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.024+0000 7f8490ff9640 3 client.19088.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:54.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.024+0000 7f8490ff9640 3 client.19088.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:54.024 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.025+0000 7f8490ff9640 20 client.19088.objecter dump_active .. 0 homeless 2023-12-11T20:49:54.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.025+0000 7f8490ff9640 1 -- 172.21.15.86:0/1886399032 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f847c054ff0 con 0x7f84940a8e00 2023-12-11T20:49:54.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.025+0000 7f8490ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:54.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.025+0000 7f8490ff9640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:54.213 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.214+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 --> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84940acfb0 con 0x7f847406c820 2023-12-11T20:49:54.219 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.220+0000 7f8490ff9640 1 -- 172.21.15.86:0/1886399032 <== mgr.4111 v2:172.21.15.133:6800/1322765564 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151897 (secure 0 0 0) 0x7f84940acfb0 con 0x7f847406c820 2023-12-11T20:49:54.221 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-11T20:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 10 librados: watch_flush enter 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 10 librados: watch_flush exit 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 20 client.19088.objecter shutdown clearing up homeless session... 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 10 client.19088.objecter successfully canceled tick 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f847406c820 msgr2=0x7f847406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f847406c820 0x7f847406ece0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8484005880 tx=0x7f8484009040 comp rx=0 tx=0).stop 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 10 monclient: shutdown 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84940a8e00 msgr2=0x7f8494135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84940a8e00 0x7f8494135710 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f847c00e9b0 tx=0x7f847c00ee70 comp rx=0 tx=0).stop 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 shutdown_connections 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7f847406c820 0x7f847406ece0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84940a8e00 0x7f8494135710 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84941423b0 0x7f84940b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 --2- 172.21.15.86:0/1886399032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84940a8a20 0x7f84940ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.222+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 >> 172.21.15.86:0/1886399032 conn(0x7f84940b7870 msgr2=0x7f84940b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.223+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 shutdown_connections 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.223+0000 7f8499ac9640 1 -- 172.21.15.86:0/1886399032 wait complete. 2023-12-11T20:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.223+0000 7f8499ac9640 1 librados: shutdown 2023-12-11T20:49:54.234 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":2055,"stamp":"2023-12-11T20:49:53.121048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":534584,"kb_used_data":17208,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754440136,"statfs":{"total":773094113280,"available":772546699264,"internally_reserved":0,"allocated":17620992,"data_stored":5245138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":529714737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002851"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1012,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:07.141741+0000","last_change":"2023-12-11T20:49:07.141741+0000","last_active":"2023-12-11T20:49:07.141741+0000","last_peered":"2023-12-11T20:49:07.141741+0000","last_clean":"2023-12-11T20:49:07.141741+0000","last_became_active":"2023-12-11T20:18:00.530681+0000","last_became_peered":"2023-12-11T20:18:00.530681+0000","last_unstale":"2023-12-11T20:49:07.141741+0000","last_undegraded":"2023-12-11T20:49:07.141741+0000","last_fullsized":"2023-12-11T20:49:07.141741+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:07.138431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:07.138431+0000","last_clean_scrub_stamp":"2023-12-11T20:49:07.138431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:43:34.623842+0000","scrub_duration":0.031828385000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:10.009088+0000","last_change":"2023-12-11T20:49:10.009088+0000","last_active":"2023-12-11T20:49:10.009088+0000","last_peered":"2023-12-11T20:49:10.009088+0000","last_clean":"2023-12-11T20:49:10.009088+0000","last_became_active":"2023-12-11T20:18:01.459740+0000","last_became_peered":"2023-12-11T20:18:01.459740+0000","last_unstale":"2023-12-11T20:49:10.009088+0000","last_undegraded":"2023-12-11T20:49:10.009088+0000","last_fullsized":"2023-12-11T20:49:10.009088+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:10.006984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:10.006984+0000","last_clean_scrub_stamp":"2023-12-11T20:49:10.006984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:40:50.627346+0000","scrub_duration":0.184697679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1020,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:14.479786+0000","last_change":"2023-12-11T20:49:14.480468+0000","last_active":"2023-12-11T20:49:14.479786+0000","last_peered":"2023-12-11T20:49:14.479786+0000","last_clean":"2023-12-11T20:49:14.479786+0000","last_became_active":"2023-12-11T20:18:11.552993+0000","last_became_peered":"2023-12-11T20:18:11.552993+0000","last_unstale":"2023-12-11T20:49:14.479786+0000","last_undegraded":"2023-12-11T20:49:14.479786+0000","last_fullsized":"2023-12-11T20:49:14.479786+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:14.478434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:14.478434+0000","last_clean_scrub_stamp":"2023-12-11T20:49:14.478434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:44:01.670384+0000","scrub_duration":0.044903595999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1015,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:18.955364+0000","last_change":"2023-12-11T20:49:18.956011+0000","last_active":"2023-12-11T20:49:18.955364+0000","last_peered":"2023-12-11T20:49:18.955364+0000","last_clean":"2023-12-11T20:49:18.955364+0000","last_became_active":"2023-12-11T20:18:04.748213+0000","last_became_peered":"2023-12-11T20:18:04.748213+0000","last_unstale":"2023-12-11T20:49:18.955364+0000","last_undegraded":"2023-12-11T20:49:18.955364+0000","last_fullsized":"2023-12-11T20:49:18.955364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:18.954003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:18.954003+0000","last_clean_scrub_stamp":"2023-12-11T20:49:18.954003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:57:36.982193+0000","scrub_duration":0.053313004999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:13.889915+0000","last_change":"2023-12-11T20:49:13.889915+0000","last_active":"2023-12-11T20:49:13.889915+0000","last_peered":"2023-12-11T20:49:13.889915+0000","last_clean":"2023-12-11T20:49:13.889915+0000","last_became_active":"2023-12-11T20:18:06.871380+0000","last_became_peered":"2023-12-11T20:18:06.871380+0000","last_unstale":"2023-12-11T20:49:13.889915+0000","last_undegraded":"2023-12-11T20:49:13.889915+0000","last_fullsized":"2023-12-11T20:49:13.889915+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:13.887611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:13.887611+0000","last_clean_scrub_stamp":"2023-12-11T20:49:13.887611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:26:11.335636+0000","scrub_duration":0.023335913,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1021,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:13.931187+0000","last_change":"2023-12-11T20:49:13.931187+0000","last_active":"2023-12-11T20:49:13.931187+0000","last_peered":"2023-12-11T20:49:13.931187+0000","last_clean":"2023-12-11T20:49:13.931187+0000","last_became_active":"2023-12-11T20:18:00.891399+0000","last_became_peered":"2023-12-11T20:18:00.891399+0000","last_unstale":"2023-12-11T20:49:13.931187+0000","last_undegraded":"2023-12-11T20:49:13.931187+0000","last_fullsized":"2023-12-11T20:49:13.931187+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:13.929236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:13.929236+0000","last_clean_scrub_stamp":"2023-12-11T20:49:13.929236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:15:44.597307+0000","scrub_duration":0.056576538000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1022,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:13.450413+0000","last_change":"2023-12-11T20:49:13.451067+0000","last_active":"2023-12-11T20:49:13.450413+0000","last_peered":"2023-12-11T20:49:13.450413+0000","last_clean":"2023-12-11T20:49:13.450413+0000","last_became_active":"2023-12-11T20:18:11.323743+0000","last_became_peered":"2023-12-11T20:18:11.323743+0000","last_unstale":"2023-12-11T20:49:13.450413+0000","last_undegraded":"2023-12-11T20:49:13.450413+0000","last_fullsized":"2023-12-11T20:49:13.450413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:13.449047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:13.449047+0000","last_clean_scrub_stamp":"2023-12-11T20:49:13.449047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:01:48.607227+0000","scrub_duration":0.045758721000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1033,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:27.416519+0000","last_change":"2023-12-11T20:49:27.417180+0000","last_active":"2023-12-11T20:49:27.416519+0000","last_peered":"2023-12-11T20:49:27.416519+0000","last_clean":"2023-12-11T20:49:27.416519+0000","last_became_active":"2023-12-11T20:18:03.798156+0000","last_became_peered":"2023-12-11T20:18:03.798156+0000","last_unstale":"2023-12-11T20:49:27.416519+0000","last_undegraded":"2023-12-11T20:49:27.416519+0000","last_fullsized":"2023-12-11T20:49:27.416519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:27.415205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:27.415205+0000","last_clean_scrub_stamp":"2023-12-11T20:49:27.415205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T00:01:34.614827+0000","scrub_duration":0.073947849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1026,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:12.947697+0000","last_change":"2023-12-11T20:49:12.948332+0000","last_active":"2023-12-11T20:49:12.947697+0000","last_peered":"2023-12-11T20:49:12.947697+0000","last_clean":"2023-12-11T20:49:12.947697+0000","last_became_active":"2023-12-11T20:18:06.509738+0000","last_became_peered":"2023-12-11T20:18:06.509738+0000","last_unstale":"2023-12-11T20:49:12.947697+0000","last_undegraded":"2023-12-11T20:49:12.947697+0000","last_fullsized":"2023-12-11T20:49:12.947697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:12.946399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:12.946399+0000","last_clean_scrub_stamp":"2023-12-11T20:49:12.946399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:44:02.649287+0000","scrub_duration":0.023431917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"1057'8","reported_seq":1037,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:21.373637+0000","last_change":"2023-12-11T20:49:21.374468+0000","last_active":"2023-12-11T20:49:21.373637+0000","last_peered":"2023-12-11T20:49:21.373637+0000","last_clean":"2023-12-11T20:49:21.373637+0000","last_became_active":"2023-12-11T20:18:05.684774+0000","last_became_peered":"2023-12-11T20:18:05.684774+0000","last_unstale":"2023-12-11T20:49:21.373637+0000","last_undegraded":"2023-12-11T20:49:21.373637+0000","last_fullsized":"2023-12-11T20:49:21.373637+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"1057'8","last_scrub_stamp":"2023-12-11T20:49:21.372297+0000","last_deep_scrub":"1057'8","last_deep_scrub_stamp":"2023-12-11T20:49:21.372297+0000","last_clean_scrub_stamp":"2023-12-11T20:49:21.372297+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:45:17.751307+0000","scrub_duration":0.080219269999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:09.868318+0000","last_change":"2023-12-11T20:49:09.868318+0000","last_active":"2023-12-11T20:49:09.868318+0000","last_peered":"2023-12-11T20:49:09.868318+0000","last_clean":"2023-12-11T20:49:09.868318+0000","last_became_active":"2023-12-11T20:18:11.576567+0000","last_became_peered":"2023-12-11T20:18:11.576567+0000","last_unstale":"2023-12-11T20:49:09.868318+0000","last_undegraded":"2023-12-11T20:49:09.868318+0000","last_fullsized":"2023-12-11T20:49:09.868318+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:09.866209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:09.866209+0000","last_clean_scrub_stamp":"2023-12-11T20:49:09.866209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:07:41.978437+0000","scrub_duration":0.034387038000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1034,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:20.485168+0000","last_change":"2023-12-11T20:49:20.485805+0000","last_active":"2023-12-11T20:49:20.485168+0000","last_peered":"2023-12-11T20:49:20.485168+0000","last_clean":"2023-12-11T20:49:20.485168+0000","last_became_active":"2023-12-11T20:18:03.237738+0000","last_became_peered":"2023-12-11T20:18:03.237738+0000","last_unstale":"2023-12-11T20:49:20.485168+0000","last_undegraded":"2023-12-11T20:49:20.485168+0000","last_fullsized":"2023-12-11T20:49:20.485168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:20.483814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:20.483814+0000","last_clean_scrub_stamp":"2023-12-11T20:49:20.483814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:41:56.135136+0000","scrub_duration":0.065074221000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1057'78","reported_seq":1133,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:08.936824+0000","last_change":"2023-12-11T20:49:08.937802+0000","last_active":"2023-12-11T20:49:08.936824+0000","last_peered":"2023-12-11T20:49:08.936824+0000","last_clean":"2023-12-11T20:49:08.936824+0000","last_became_active":"2023-12-11T20:16:49.435576+0000","last_became_peered":"2023-12-11T20:16:49.435576+0000","last_unstale":"2023-12-11T20:49:08.936824+0000","last_undegraded":"2023-12-11T20:49:08.936824+0000","last_fullsized":"2023-12-11T20:49:08.936824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"1057'78","last_scrub_stamp":"2023-12-11T20:49:08.935520+0000","last_deep_scrub":"1057'78","last_deep_scrub_stamp":"2023-12-11T20:49:08.935520+0000","last_clean_scrub_stamp":"2023-12-11T20:49:08.935520+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:34:51.152246+0000","scrub_duration":0.11416854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:11.079430+0000","last_change":"2023-12-11T20:49:11.080091+0000","last_active":"2023-12-11T20:49:11.079430+0000","last_peered":"2023-12-11T20:49:11.079430+0000","last_clean":"2023-12-11T20:49:11.079430+0000","last_became_active":"2023-12-11T20:16:58.638544+0000","last_became_peered":"2023-12-11T20:16:58.638544+0000","last_unstale":"2023-12-11T20:49:11.079430+0000","last_undegraded":"2023-12-11T20:49:11.079430+0000","last_fullsized":"2023-12-11T20:49:11.079430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:11.078127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:11.078127+0000","last_clean_scrub_stamp":"2023-12-11T20:49:11.078127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:20:31.188162+0000","scrub_duration":0.025714267999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1013,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:14.181425+0000","last_change":"2023-12-11T20:49:14.182122+0000","last_active":"2023-12-11T20:49:14.181425+0000","last_peered":"2023-12-11T20:49:14.181425+0000","last_clean":"2023-12-11T20:49:14.181425+0000","last_became_active":"2023-12-11T20:18:12.924715+0000","last_became_peered":"2023-12-11T20:18:12.924715+0000","last_unstale":"2023-12-11T20:49:14.181425+0000","last_undegraded":"2023-12-11T20:49:14.181425+0000","last_fullsized":"2023-12-11T20:49:14.181425+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:14.180019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:14.180019+0000","last_clean_scrub_stamp":"2023-12-11T20:49:14.180019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:16:50.154388+0000","scrub_duration":0.054111566999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:10.334120+0000","last_change":"2023-12-11T20:49:10.338763+0000","last_active":"2023-12-11T20:49:10.334120+0000","last_peered":"2023-12-11T20:49:10.334120+0000","last_clean":"2023-12-11T20:49:10.334120+0000","last_became_active":"2023-12-11T20:16:58.008642+0000","last_became_peered":"2023-12-11T20:16:58.008642+0000","last_unstale":"2023-12-11T20:49:10.334120+0000","last_undegraded":"2023-12-11T20:49:10.334120+0000","last_fullsized":"2023-12-11T20:49:10.334120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:10.330090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:10.330090+0000","last_clean_scrub_stamp":"2023-12-11T20:49:10.330090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:05:06.782303+0000","scrub_duration":0.21427059700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1026,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:22.961515+0000","last_change":"2023-12-11T20:49:22.962187+0000","last_active":"2023-12-11T20:49:22.961515+0000","last_peered":"2023-12-11T20:49:22.961515+0000","last_clean":"2023-12-11T20:49:22.961515+0000","last_became_active":"2023-12-11T20:18:07.135656+0000","last_became_peered":"2023-12-11T20:18:07.135656+0000","last_unstale":"2023-12-11T20:49:22.961515+0000","last_undegraded":"2023-12-11T20:49:22.961515+0000","last_fullsized":"2023-12-11T20:49:22.961515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:22.960080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:22.960080+0000","last_clean_scrub_stamp":"2023-12-11T20:49:22.960080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:41:09.238717+0000","scrub_duration":0.029403362999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1038,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:13.509279+0000","last_change":"2023-12-11T20:49:13.509933+0000","last_active":"2023-12-11T20:49:13.509279+0000","last_peered":"2023-12-11T20:49:13.509279+0000","last_clean":"2023-12-11T20:49:13.509279+0000","last_became_active":"2023-12-11T20:16:58.208987+0000","last_became_peered":"2023-12-11T20:16:58.208987+0000","last_unstale":"2023-12-11T20:49:13.509279+0000","last_undegraded":"2023-12-11T20:49:13.509279+0000","last_fullsized":"2023-12-11T20:49:13.509279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:13.507891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:13.507891+0000","last_clean_scrub_stamp":"2023-12-11T20:49:13.507891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:18:30.657122+0000","scrub_duration":0.026209349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1034,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:15.379092+0000","last_change":"2023-12-11T20:49:15.379092+0000","last_active":"2023-12-11T20:49:15.379092+0000","last_peered":"2023-12-11T20:49:15.379092+0000","last_clean":"2023-12-11T20:49:15.379092+0000","last_became_active":"2023-12-11T20:18:01.354083+0000","last_became_peered":"2023-12-11T20:18:01.354083+0000","last_unstale":"2023-12-11T20:49:15.379092+0000","last_undegraded":"2023-12-11T20:49:15.379092+0000","last_fullsized":"2023-12-11T20:49:15.379092+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:15.377081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:15.377081+0000","last_clean_scrub_stamp":"2023-12-11T20:49:15.377081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:10:33.165194+0000","scrub_duration":0.080873467000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:18.070420+0000","last_change":"2023-12-11T20:49:18.071089+0000","last_active":"2023-12-11T20:49:18.070420+0000","last_peered":"2023-12-11T20:49:18.070420+0000","last_clean":"2023-12-11T20:49:18.070420+0000","last_became_active":"2023-12-11T20:16:58.349599+0000","last_became_peered":"2023-12-11T20:16:58.349599+0000","last_unstale":"2023-12-11T20:49:18.070420+0000","last_undegraded":"2023-12-11T20:49:18.070420+0000","last_fullsized":"2023-12-11T20:49:18.070420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:18.069072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:18.069072+0000","last_clean_scrub_stamp":"2023-12-11T20:49:18.069072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:26:02.233542+0000","scrub_duration":0.025866693999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:10.837573+0000","last_change":"2023-12-11T20:49:10.838821+0000","last_active":"2023-12-11T20:49:10.837573+0000","last_peered":"2023-12-11T20:49:10.837573+0000","last_clean":"2023-12-11T20:49:10.837573+0000","last_became_active":"2023-12-11T20:18:02.117789+0000","last_became_peered":"2023-12-11T20:18:02.117789+0000","last_unstale":"2023-12-11T20:49:10.837573+0000","last_undegraded":"2023-12-11T20:49:10.837573+0000","last_fullsized":"2023-12-11T20:49:10.837573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:10.835332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:10.835332+0000","last_clean_scrub_stamp":"2023-12-11T20:49:10.835332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:14:03.415524+0000","scrub_duration":0.045176734000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1031,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:10.958036+0000","last_change":"2023-12-11T20:49:10.958716+0000","last_active":"2023-12-11T20:49:10.958036+0000","last_peered":"2023-12-11T20:49:10.958036+0000","last_clean":"2023-12-11T20:49:10.958036+0000","last_became_active":"2023-12-11T20:16:58.383357+0000","last_became_peered":"2023-12-11T20:16:58.383357+0000","last_unstale":"2023-12-11T20:49:10.958036+0000","last_undegraded":"2023-12-11T20:49:10.958036+0000","last_fullsized":"2023-12-11T20:49:10.958036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:10.955322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:10.955322+0000","last_clean_scrub_stamp":"2023-12-11T20:49:10.955322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:37:08.717834+0000","scrub_duration":0.026970309000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1026,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:33.063385+0000","last_change":"2023-12-11T20:49:33.063385+0000","last_active":"2023-12-11T20:49:33.063385+0000","last_peered":"2023-12-11T20:49:33.063385+0000","last_clean":"2023-12-11T20:49:33.063385+0000","last_became_active":"2023-12-11T20:18:13.700424+0000","last_became_peered":"2023-12-11T20:18:13.700424+0000","last_unstale":"2023-12-11T20:49:33.063385+0000","last_undegraded":"2023-12-11T20:49:33.063385+0000","last_fullsized":"2023-12-11T20:49:33.063385+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:33.061444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:33.061444+0000","last_clean_scrub_stamp":"2023-12-11T20:49:33.061444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:06:23.683451+0000","scrub_duration":0.054160918000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:23.074066+0000","last_change":"2023-12-11T20:49:23.074747+0000","last_active":"2023-12-11T20:49:23.074066+0000","last_peered":"2023-12-11T20:49:23.074066+0000","last_clean":"2023-12-11T20:49:23.074066+0000","last_became_active":"2023-12-11T20:16:58.674889+0000","last_became_peered":"2023-12-11T20:16:58.674889+0000","last_unstale":"2023-12-11T20:49:23.074066+0000","last_undegraded":"2023-12-11T20:49:23.074066+0000","last_fullsized":"2023-12-11T20:49:23.074066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:23.072765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:23.072765+0000","last_clean_scrub_stamp":"2023-12-11T20:49:23.072765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:37:27.674302+0000","scrub_duration":0.024438919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1033,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:24.456293+0000","last_change":"2023-12-11T20:49:24.456976+0000","last_active":"2023-12-11T20:49:24.456293+0000","last_peered":"2023-12-11T20:49:24.456293+0000","last_clean":"2023-12-11T20:49:24.456293+0000","last_became_active":"2023-12-11T20:18:03.180691+0000","last_became_peered":"2023-12-11T20:18:03.180691+0000","last_unstale":"2023-12-11T20:49:24.456293+0000","last_undegraded":"2023-12-11T20:49:24.456293+0000","last_fullsized":"2023-12-11T20:49:24.456293+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:24.454965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:24.454965+0000","last_clean_scrub_stamp":"2023-12-11T20:49:24.454965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:36:06.006630+0000","scrub_duration":0.080620827000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1039,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:11.305527+0000","last_change":"2023-12-11T20:49:11.305527+0000","last_active":"2023-12-11T20:49:11.305527+0000","last_peered":"2023-12-11T20:49:11.305527+0000","last_clean":"2023-12-11T20:49:11.305527+0000","last_became_active":"2023-12-11T20:16:57.255904+0000","last_became_peered":"2023-12-11T20:16:57.255904+0000","last_unstale":"2023-12-11T20:49:11.305527+0000","last_undegraded":"2023-12-11T20:49:11.305527+0000","last_fullsized":"2023-12-11T20:49:11.305527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:11.303567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:11.303567+0000","last_clean_scrub_stamp":"2023-12-11T20:49:11.303567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:47:39.399616+0000","scrub_duration":0.024613408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1026,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:23.950534+0000","last_change":"2023-12-11T20:49:23.950534+0000","last_active":"2023-12-11T20:49:23.950534+0000","last_peered":"2023-12-11T20:49:23.950534+0000","last_clean":"2023-12-11T20:49:23.950534+0000","last_became_active":"2023-12-11T20:18:33.762733+0000","last_became_peered":"2023-12-11T20:18:33.762733+0000","last_unstale":"2023-12-11T20:49:23.950534+0000","last_undegraded":"2023-12-11T20:49:23.950534+0000","last_fullsized":"2023-12-11T20:49:23.950534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:23.948571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:23.948571+0000","last_clean_scrub_stamp":"2023-12-11T20:49:23.948571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:11:37.564191+0000","scrub_duration":0.028664346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1023,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:14.121459+0000","last_change":"2023-12-11T20:49:14.121459+0000","last_active":"2023-12-11T20:49:14.121459+0000","last_peered":"2023-12-11T20:49:14.121459+0000","last_clean":"2023-12-11T20:49:14.121459+0000","last_became_active":"2023-12-11T20:16:58.071647+0000","last_became_peered":"2023-12-11T20:16:58.071647+0000","last_unstale":"2023-12-11T20:49:14.121459+0000","last_undegraded":"2023-12-11T20:49:14.121459+0000","last_fullsized":"2023-12-11T20:49:14.121459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:14.118031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:14.118031+0000","last_clean_scrub_stamp":"2023-12-11T20:49:14.118031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:10:43.276760+0000","scrub_duration":0.031451593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1017,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:19.970759+0000","last_change":"2023-12-11T20:49:19.970759+0000","last_active":"2023-12-11T20:49:19.970759+0000","last_peered":"2023-12-11T20:49:19.970759+0000","last_clean":"2023-12-11T20:49:19.970759+0000","last_became_active":"2023-12-11T20:18:11.885388+0000","last_became_peered":"2023-12-11T20:18:11.885388+0000","last_unstale":"2023-12-11T20:49:19.970759+0000","last_undegraded":"2023-12-11T20:49:19.970759+0000","last_fullsized":"2023-12-11T20:49:19.970759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:19.968634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:19.968634+0000","last_clean_scrub_stamp":"2023-12-11T20:49:19.968634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T00:21:57.498718+0000","scrub_duration":0.073009713000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1035,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:17.408654+0000","last_change":"2023-12-11T20:49:17.409350+0000","last_active":"2023-12-11T20:49:17.408654+0000","last_peered":"2023-12-11T20:49:17.408654+0000","last_clean":"2023-12-11T20:49:17.408654+0000","last_became_active":"2023-12-11T20:18:12.786107+0000","last_became_peered":"2023-12-11T20:18:12.786107+0000","last_unstale":"2023-12-11T20:49:17.408654+0000","last_undegraded":"2023-12-11T20:49:17.408654+0000","last_fullsized":"2023-12-11T20:49:17.408654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:17.407244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:17.407244+0000","last_clean_scrub_stamp":"2023-12-11T20:49:17.407244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:05:03.190257+0000","scrub_duration":0.14288842400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1034,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:23.447118+0000","last_change":"2023-12-11T20:49:23.447754+0000","last_active":"2023-12-11T20:49:23.447118+0000","last_peered":"2023-12-11T20:49:23.447118+0000","last_clean":"2023-12-11T20:49:23.447118+0000","last_became_active":"2023-12-11T20:18:00.469176+0000","last_became_peered":"2023-12-11T20:18:00.469176+0000","last_unstale":"2023-12-11T20:49:23.447118+0000","last_undegraded":"2023-12-11T20:49:23.447118+0000","last_fullsized":"2023-12-11T20:49:23.447118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:23.445559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:23.445559+0000","last_clean_scrub_stamp":"2023-12-11T20:49:23.445559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:32:20.460753+0000","scrub_duration":0.041653294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1020,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:15.475070+0000","last_change":"2023-12-11T20:49:15.475942+0000","last_active":"2023-12-11T20:49:15.475070+0000","last_peered":"2023-12-11T20:49:15.475070+0000","last_clean":"2023-12-11T20:49:15.475070+0000","last_became_active":"2023-12-11T20:18:05.138035+0000","last_became_peered":"2023-12-11T20:18:05.138035+0000","last_unstale":"2023-12-11T20:49:15.475070+0000","last_undegraded":"2023-12-11T20:49:15.475070+0000","last_fullsized":"2023-12-11T20:49:15.475070+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:15.473598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:15.473598+0000","last_clean_scrub_stamp":"2023-12-11T20:49:15.473598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T22:05:17.067244+0000","scrub_duration":0.049786271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1016,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:31.196507+0000","last_change":"2023-12-11T20:49:31.197160+0000","last_active":"2023-12-11T20:49:31.196507+0000","last_peered":"2023-12-11T20:49:31.196507+0000","last_clean":"2023-12-11T20:49:31.196507+0000","last_became_active":"2023-12-11T20:18:10.094694+0000","last_became_peered":"2023-12-11T20:18:10.094694+0000","last_unstale":"2023-12-11T20:49:31.196507+0000","last_undegraded":"2023-12-11T20:49:31.196507+0000","last_fullsized":"2023-12-11T20:49:31.196507+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:31.195180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:31.195180+0000","last_clean_scrub_stamp":"2023-12-11T20:49:31.195180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:53:30.112343+0000","scrub_duration":0.031544593000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1024,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:14.993451+0000","last_change":"2023-12-11T20:49:14.994122+0000","last_active":"2023-12-11T20:49:14.993451+0000","last_peered":"2023-12-11T20:49:14.993451+0000","last_clean":"2023-12-11T20:49:14.993451+0000","last_became_active":"2023-12-11T20:18:07.062338+0000","last_became_peered":"2023-12-11T20:18:07.062338+0000","last_unstale":"2023-12-11T20:49:14.993451+0000","last_undegraded":"2023-12-11T20:49:14.993451+0000","last_fullsized":"2023-12-11T20:49:14.993451+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:14.992072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:14.992072+0000","last_clean_scrub_stamp":"2023-12-11T20:49:14.992072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:44:46.471306+0000","scrub_duration":0.089570364999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1017,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:32.187478+0000","last_change":"2023-12-11T20:49:32.188143+0000","last_active":"2023-12-11T20:49:32.187478+0000","last_peered":"2023-12-11T20:49:32.187478+0000","last_clean":"2023-12-11T20:49:32.187478+0000","last_became_active":"2023-12-11T20:18:19.665316+0000","last_became_peered":"2023-12-11T20:18:19.665316+0000","last_unstale":"2023-12-11T20:49:32.187478+0000","last_undegraded":"2023-12-11T20:49:32.187478+0000","last_fullsized":"2023-12-11T20:49:32.187478+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:32.186117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:32.186117+0000","last_clean_scrub_stamp":"2023-12-11T20:49:32.186117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:00:25.245373+0000","scrub_duration":0.043563756000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1017,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:26.176549+0000","last_change":"2023-12-11T20:49:26.177232+0000","last_active":"2023-12-11T20:49:26.176549+0000","last_peered":"2023-12-11T20:49:26.176549+0000","last_clean":"2023-12-11T20:49:26.176549+0000","last_became_active":"2023-12-11T20:18:18.557249+0000","last_became_peered":"2023-12-11T20:18:18.557249+0000","last_unstale":"2023-12-11T20:49:26.176549+0000","last_undegraded":"2023-12-11T20:49:26.176549+0000","last_fullsized":"2023-12-11T20:49:26.176549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:26.175055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:26.175055+0000","last_clean_scrub_stamp":"2023-12-11T20:49:26.175055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:28:10.873713+0000","scrub_duration":0.063818097000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1016,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:25.100719+0000","last_change":"2023-12-11T20:49:25.101410+0000","last_active":"2023-12-11T20:49:25.100719+0000","last_peered":"2023-12-11T20:49:25.100719+0000","last_clean":"2023-12-11T20:49:25.100719+0000","last_became_active":"2023-12-11T20:18:19.685187+0000","last_became_peered":"2023-12-11T20:18:19.685187+0000","last_unstale":"2023-12-11T20:49:25.100719+0000","last_undegraded":"2023-12-11T20:49:25.100719+0000","last_fullsized":"2023-12-11T20:49:25.100719+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:25.099287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:25.099287+0000","last_clean_scrub_stamp":"2023-12-11T20:49:25.099287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:41:03.663936+0000","scrub_duration":0.029382286000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1019,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:15.998742+0000","last_change":"2023-12-11T20:49:15.998742+0000","last_active":"2023-12-11T20:49:15.998742+0000","last_peered":"2023-12-11T20:49:15.998742+0000","last_clean":"2023-12-11T20:49:15.998742+0000","last_became_active":"2023-12-11T20:18:00.554682+0000","last_became_peered":"2023-12-11T20:18:00.554682+0000","last_unstale":"2023-12-11T20:49:15.998742+0000","last_undegraded":"2023-12-11T20:49:15.998742+0000","last_fullsized":"2023-12-11T20:49:15.998742+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:15.996775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:15.996775+0000","last_clean_scrub_stamp":"2023-12-11T20:49:15.996775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:54:56.936253+0000","scrub_duration":0.075660405999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1025,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:30.042353+0000","last_change":"2023-12-11T20:49:30.042353+0000","last_active":"2023-12-11T20:49:30.042353+0000","last_peered":"2023-12-11T20:49:30.042353+0000","last_clean":"2023-12-11T20:49:30.042353+0000","last_became_active":"2023-12-11T20:18:13.689623+0000","last_became_peered":"2023-12-11T20:18:13.689623+0000","last_unstale":"2023-12-11T20:49:30.042353+0000","last_undegraded":"2023-12-11T20:49:30.042353+0000","last_fullsized":"2023-12-11T20:49:30.042353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:30.040447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:30.040447+0000","last_clean_scrub_stamp":"2023-12-11T20:49:30.040447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T01:57:12.232772+0000","scrub_duration":0.043914570999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1017,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:35.238836+0000","last_change":"2023-12-11T20:49:35.239578+0000","last_active":"2023-12-11T20:49:35.238836+0000","last_peered":"2023-12-11T20:49:35.238836+0000","last_clean":"2023-12-11T20:49:35.238836+0000","last_became_active":"2023-12-11T20:18:10.112911+0000","last_became_peered":"2023-12-11T20:18:10.112911+0000","last_unstale":"2023-12-11T20:49:35.238836+0000","last_undegraded":"2023-12-11T20:49:35.238836+0000","last_fullsized":"2023-12-11T20:49:35.238836+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:35.237404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:35.237404+0000","last_clean_scrub_stamp":"2023-12-11T20:49:35.237404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:00:23.211724+0000","scrub_duration":0.070226988000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1011,"reported_epoch":1057,"state":"active+clean","last_fresh":"2023-12-11T20:49:11.152306+0000","last_change":"2023-12-11T20:49:11.153004+0000","last_active":"2023-12-11T20:49:11.152306+0000","last_peered":"2023-12-11T20:49:11.152306+0000","last_clean":"2023-12-11T20:49:11.152306+0000","last_became_active":"2023-12-11T20:18:13.203517+0000","last_became_peered":"2023-12-11T20:18:13.203517+0000","last_unstale":"2023-12-11T20:49:11.152306+0000","last_undegraded":"2023-12-11T20:49:11.152306+0000","last_fullsized":"2023-12-11T20:49:11.152306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-11T20:49:11.150943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-11T20:49:11.150943+0000","last_clean_scrub_stamp":"2023-12-11T20:49:11.150943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T21:52:36.723162+0000","scrub_duration":0.026594514999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719477133,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69328,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302512,"statfs":{"total":96636764160,"available":96565772288,"internally_reserved":0,"allocated":2105344,"data_stored":556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":17.427,"5min":10.535,"15min":14.893000000000001},"min":{"1min":4.5739999999999998,"5min":4.5739999999999998,"15min":4.2510000000000003},"max":{"1min":112.848,"5min":112.848,"15min":221.21299999999999},"last":15.993},{"interface":"front","average":{"1min":17.669,"5min":11.170999999999999,"15min":14.507999999999999},"min":{"1min":6.2409999999999997,"5min":5.3739999999999997,"15min":4.7930000000000001},"max":{"1min":94.400999999999996,"5min":94.400999999999996,"15min":164.01900000000001},"last":12.092000000000001}]},{"osd":1,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":21.391999999999999,"5min":12.878,"15min":16.920999999999999},"min":{"1min":7.6660000000000004,"5min":6.8419999999999996,"15min":5.0380000000000003},"max":{"1min":117.63,"5min":117.63,"15min":195.87200000000001},"last":13.720000000000001},{"interface":"front","average":{"1min":26.209,"5min":17.108000000000001,"15min":20.576000000000001},"min":{"1min":10.843999999999999,"5min":9.6649999999999991,"15min":6.0300000000000002},"max":{"1min":118.309,"5min":118.309,"15min":190.53899999999999},"last":16.823}]},{"osd":2,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":15.789999999999999,"5min":11.457000000000001,"15min":18.841999999999999},"min":{"1min":6.4089999999999998,"5min":6.4089999999999998,"15min":5.8259999999999996},"max":{"1min":67.793000000000006,"5min":67.793000000000006,"15min":282.79000000000002},"last":7.923},{"interface":"front","average":{"1min":21.603000000000002,"5min":15.961,"15min":21.614000000000001},"min":{"1min":8.1829999999999998,"5min":8.1829999999999998,"15min":8.0570000000000004},"max":{"1min":68.638999999999996,"5min":68.638999999999996,"15min":200.148},"last":12.917}]},{"osd":3,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.715,"5min":15.484999999999999,"15min":19.199999999999999},"min":{"1min":9.6660000000000004,"5min":6.5270000000000001,"15min":5.3029999999999999},"max":{"1min":113.89400000000001,"5min":113.89400000000001,"15min":206.11799999999999},"last":10.308999999999999},{"interface":"front","average":{"1min":22.518000000000001,"5min":14.561,"15min":20.105},"min":{"1min":6.7809999999999997,"5min":6.7809999999999997,"15min":6.7809999999999997},"max":{"1min":140.172,"5min":140.172,"15min":171.63},"last":18.652999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":24.693000000000001,"5min":16.344999999999999,"15min":18.094000000000001},"min":{"1min":8.5180000000000007,"5min":7.6289999999999996,"15min":6.577},"max":{"1min":110.98399999999999,"5min":110.98399999999999,"15min":188.26599999999999},"last":14.417},{"interface":"front","average":{"1min":25.422999999999998,"5min":17.34,"15min":21.085000000000001},"min":{"1min":7.9279999999999999,"5min":7.8470000000000004,"15min":7.5300000000000002},"max":{"1min":144.084,"5min":144.084,"15min":172.57400000000001},"last":17.914000000000001}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":19.388999999999999,"5min":14.335000000000001,"15min":18.338999999999999},"min":{"1min":8.4640000000000004,"5min":7.4800000000000004,"15min":6.141},"max":{"1min":62.637,"5min":62.637,"15min":201.25399999999999},"last":10.680999999999999},{"interface":"front","average":{"1min":24.920999999999999,"5min":17.582999999999998,"15min":21.760000000000002},"min":{"1min":10.914999999999999,"5min":8.9529999999999994,"15min":8.423},"max":{"1min":92.804000000000002,"5min":92.804000000000002,"15min":200.60599999999999},"last":20.763000000000002}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":23.809999999999999,"5min":18.367999999999999,"15min":21.443000000000001},"min":{"1min":10.18,"5min":8.4909999999999997,"15min":7.899},"max":{"1min":76.634,"5min":76.634,"15min":207.72200000000001},"last":11.737},{"interface":"front","average":{"1min":22.077000000000002,"5min":17.238,"15min":22.446999999999999},"min":{"1min":9.6129999999999995,"5min":9.3019999999999996,"15min":9.3019999999999996},"max":{"1min":69.144000000000005,"5min":69.144000000000005,"15min":193.43899999999999},"last":19.869}]}]},{"osd":3,"up_from":16,"seq":68719477133,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:20 2023","interfaces":[{"interface":"back","average":{"1min":25.492999999999999,"5min":15.124000000000001,"15min":16.145},"min":{"1min":4.9790000000000001,"5min":4.4720000000000004,"15min":4.1600000000000001},"max":{"1min":230.328,"5min":230.328,"15min":266.75400000000002},"last":9.7680000000000007},{"interface":"front","average":{"1min":20.965,"5min":14.555999999999999,"15min":17.283000000000001},"min":{"1min":5.4960000000000004,"5min":5.4100000000000001,"15min":4.6619999999999999},"max":{"1min":106.741,"5min":110.068,"15min":228.12899999999999},"last":7.1559999999999997}]},{"osd":1,"last update":"Mon Dec 11 20:49:40 2023","interfaces":[{"interface":"back","average":{"1min":11.605,"5min":17.616,"15min":16.449999999999999},"min":{"1min":7.8140000000000001,"5min":6.7489999999999997,"15min":5.6180000000000003},"max":{"1min":28.449000000000002,"5min":170.31200000000001,"15min":179.11000000000001},"last":8.3789999999999996},{"interface":"front","average":{"1min":14.074,"5min":19.800999999999998,"15min":18.475000000000001},"min":{"1min":9.6300000000000008,"5min":7.0529999999999999,"15min":6.508},"max":{"1min":32.317999999999998,"5min":215.578,"15min":215.578},"last":11.599}]},{"osd":2,"last update":"Mon Dec 11 20:49:20 2023","interfaces":[{"interface":"back","average":{"1min":32.378,"5min":19.553000000000001,"15min":20.100999999999999},"min":{"1min":9.423,"5min":8.3559999999999999,"15min":7.1150000000000002},"max":{"1min":232.36199999999999,"5min":232.36199999999999,"15min":232.36199999999999},"last":11.039},{"interface":"front","average":{"1min":41.271999999999998,"5min":23.105,"15min":23.425000000000001},"min":{"1min":10.132,"5min":10.132,"15min":9.5990000000000002},"max":{"1min":239.23599999999999,"5min":239.23599999999999,"15min":239.23599999999999},"last":12.206}]},{"osd":4,"last update":"Mon Dec 11 20:49:14 2023","interfaces":[{"interface":"back","average":{"1min":49.526000000000003,"5min":24.190999999999999,"15min":22.186},"min":{"1min":11.305,"5min":10.401999999999999,"15min":9.6539999999999999},"max":{"1min":258.495,"5min":258.495,"15min":258.495},"last":12.762},{"interface":"front","average":{"1min":27.492000000000001,"5min":16.739000000000001,"15min":16.282},"min":{"1min":8.2539999999999996,"5min":7.0510000000000002,"15min":7.0510000000000002},"max":{"1min":114.783,"5min":114.783,"15min":173.66800000000001},"last":10.079000000000001}]},{"osd":5,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":40.828000000000003,"5min":19.824000000000002,"15min":23.663},"min":{"1min":10.592000000000001,"5min":7.5250000000000004,"15min":7.5250000000000004},"max":{"1min":119.569,"5min":119.569,"15min":246.55000000000001},"last":16.300999999999998},{"interface":"front","average":{"1min":49.973999999999997,"5min":16.795999999999999,"15min":17.120999999999999},"min":{"1min":6.2610000000000001,"5min":5.4429999999999996,"15min":5.4429999999999996},"max":{"1min":203.21799999999999,"5min":203.21799999999999,"15min":205.15799999999999},"last":6.3079999999999998}]},{"osd":6,"last update":"Mon Dec 11 20:49:14 2023","interfaces":[{"interface":"back","average":{"1min":59.798999999999999,"5min":27.696999999999999,"15min":27.488},"min":{"1min":12.551,"5min":10.831,"15min":9.9320000000000004},"max":{"1min":285.733,"5min":285.733,"15min":285.733},"last":14.010999999999999},{"interface":"front","average":{"1min":39.789999999999999,"5min":22.498999999999999,"15min":24.436},"min":{"1min":12.676,"5min":9.8339999999999996,"15min":8.391},"max":{"1min":171.57499999999999,"5min":171.57499999999999,"15min":208.00299999999999},"last":14.567}]},{"osd":7,"last update":"Mon Dec 11 20:48:51 2023","interfaces":[{"interface":"back","average":{"1min":60.445,"5min":26.123000000000001,"15min":21.013999999999999},"min":{"1min":12.372,"5min":12.295999999999999,"15min":6.4690000000000003},"max":{"1min":260.45999999999998,"5min":260.45999999999998,"15min":260.45999999999998},"last":15.130000000000001},{"interface":"front","average":{"1min":44.287999999999997,"5min":22.015999999999998,"15min":25.881},"min":{"1min":12.273999999999999,"5min":10.863,"15min":10.863},"max":{"1min":172.559,"5min":172.559,"15min":264.99299999999999},"last":17.085000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477132,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":20.559000000000001,"5min":12.036,"15min":15.512},"min":{"1min":5.0780000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"max":{"1min":89.881,"5min":89.881,"15min":200.447},"last":12.725},{"interface":"front","average":{"1min":19.225000000000001,"5min":11.49,"15min":16.253},"min":{"1min":5.6619999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":92.114000000000004,"5min":92.114000000000004,"15min":198.22900000000001},"last":10.238}]},{"osd":1,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":21.713000000000001,"5min":13.699,"15min":16.245999999999999},"min":{"1min":7.0590000000000002,"5min":5.7869999999999999,"15min":5.7869999999999999},"max":{"1min":89.183999999999997,"5min":89.183999999999997,"15min":143.56},"last":8.3089999999999993},{"interface":"front","average":{"1min":27.739999999999998,"5min":15.618,"15min":17.497},"min":{"1min":6.1879999999999997,"5min":6.1879999999999997,"15min":6.1879999999999997},"max":{"1min":175.16200000000001,"5min":175.16200000000001,"15min":175.16200000000001},"last":11.442}]},{"osd":3,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":19.030999999999999,"5min":11.109999999999999,"15min":13.959},"min":{"1min":6.0780000000000003,"5min":5.7960000000000003,"15min":5.3550000000000004},"max":{"1min":99.146000000000001,"5min":99.146000000000001,"15min":171.71700000000001},"last":8.9030000000000005},{"interface":"front","average":{"1min":21.315999999999999,"5min":15.250999999999999,"15min":17.931000000000001},"min":{"1min":8.6479999999999997,"5min":8.0220000000000002,"15min":5.9749999999999996},"max":{"1min":82.924999999999997,"5min":82.924999999999997,"15min":182.63399999999999},"last":16.998999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":32.999000000000002,"5min":17.382999999999999,"15min":19.341999999999999},"min":{"1min":6.694,"5min":6.694,"15min":6.694},"max":{"1min":192.851,"5min":192.851,"15min":192.851},"last":12.108000000000001},{"interface":"front","average":{"1min":40.692999999999998,"5min":21.776,"15min":24.178999999999998},"min":{"1min":11.397,"5min":10.923999999999999,"15min":10.285},"max":{"1min":259.05900000000003,"5min":259.05900000000003,"15min":259.05900000000003},"last":13.98}]},{"osd":5,"last update":"Mon Dec 11 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":49.709000000000003,"5min":22.446999999999999,"15min":21.960999999999999},"min":{"1min":7.8159999999999998,"5min":7.8159999999999998,"15min":7.8159999999999998},"max":{"1min":364.11200000000002,"5min":364.11200000000002,"15min":364.11200000000002},"last":13.352},{"interface":"front","average":{"1min":39.561999999999998,"5min":20.593,"15min":24.591999999999999},"min":{"1min":10.625999999999999,"5min":9.1349999999999998,"15min":8.5020000000000007},"max":{"1min":296.74000000000001,"5min":296.74000000000001,"15min":296.74000000000001},"last":15.52}]},{"osd":6,"last update":"Mon Dec 11 20:49:22 2023","interfaces":[{"interface":"back","average":{"1min":18.748000000000001,"5min":15.739000000000001,"15min":17.251999999999999},"min":{"1min":8.0180000000000007,"5min":6.6799999999999997,"15min":6.3259999999999996},"max":{"1min":91.569999999999993,"5min":251.19300000000001,"15min":251.19300000000001},"last":17.695},{"interface":"front","average":{"1min":26.641999999999999,"5min":18.463999999999999,"15min":21.571999999999999},"min":{"1min":11.819000000000001,"5min":9.4960000000000004,"15min":9.4469999999999992},"max":{"1min":103.833,"5min":103.833,"15min":181.96299999999999},"last":19.097999999999999}]},{"osd":7,"last update":"Mon Dec 11 20:49:08 2023","interfaces":[{"interface":"back","average":{"1min":38.201999999999998,"5min":22.239999999999998,"15min":24.962},"min":{"1min":13.659000000000001,"5min":10.379,"15min":10.379},"max":{"1min":156.934,"5min":156.934,"15min":197.27199999999999},"last":19.911999999999999},{"interface":"front","average":{"1min":31.547999999999998,"5min":17.236000000000001,"15min":19.617000000000001},"min":{"1min":9.9179999999999993,"5min":8.0199999999999996,"15min":7.085},"max":{"1min":173.654,"5min":173.654,"15min":233.57400000000001},"last":16.384}]}]},{"osd":1,"up_from":16,"seq":68719477133,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69716,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302124,"statfs":{"total":96636764160,"available":96565374976,"internally_reserved":0,"allocated":2502656,"data_stored":954021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":7.8470000000000004,"15min":13.646000000000001},"min":{"1min":4.6870000000000003,"5min":4.0949999999999998,"15min":3.903},"max":{"1min":72.888000000000005,"5min":72.888000000000005,"15min":283.91899999999998},"last":6.1539999999999999},{"interface":"front","average":{"1min":17.266999999999999,"5min":9.9909999999999997,"15min":14.855},"min":{"1min":5.8250000000000002,"5min":4.9340000000000002,"15min":4.6879999999999997},"max":{"1min":99.680000000000007,"5min":99.680000000000007,"15min":391.19999999999999},"last":9.3059999999999992}]},{"osd":2,"last update":"Mon Dec 11 20:49:45 2023","interfaces":[{"interface":"back","average":{"1min":14.047000000000001,"5min":13.797000000000001,"15min":17.786999999999999},"min":{"1min":8.6199999999999992,"5min":6.0300000000000002,"15min":6.0300000000000002},"max":{"1min":24.382999999999999,"5min":84.167000000000002,"15min":279.38099999999997},"last":12.661},{"interface":"front","average":{"1min":15.75,"5min":15.313000000000001,"15min":18.920000000000002},"min":{"1min":9.7769999999999992,"5min":9.2590000000000003,"15min":6.7670000000000003},"max":{"1min":25.024999999999999,"5min":100.836,"15min":196.13200000000001},"last":13.539999999999999}]},{"osd":3,"last update":"Mon Dec 11 20:49:45 2023","interfaces":[{"interface":"back","average":{"1min":10.877000000000001,"5min":12.313000000000001,"15min":21.100999999999999},"min":{"1min":6.5869999999999997,"5min":5.7610000000000001,"15min":5.7610000000000001},"max":{"1min":24.378,"5min":101.74299999999999,"15min":346.85500000000002},"last":7.2610000000000001},{"interface":"front","average":{"1min":17.84,"5min":17.838000000000001,"15min":18.832999999999998},"min":{"1min":11.096,"5min":8.7159999999999993,"15min":5.7220000000000004},"max":{"1min":27.440999999999999,"5min":175.047,"15min":229.03299999999999},"last":14.295999999999999}]},{"osd":4,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":12.462999999999999,"15min":18.878},"min":{"1min":6.6740000000000004,"5min":6.1630000000000003,"15min":5.952},"max":{"1min":120.863,"5min":120.863,"15min":342.28199999999998},"last":8.7639999999999993},{"interface":"front","average":{"1min":33.405000000000001,"5min":18.756,"15min":23.010000000000002},"min":{"1min":12.538,"5min":10.499000000000001,"15min":6.7039999999999997},"max":{"1min":187.15100000000001,"5min":187.15100000000001,"15min":289.68799999999999},"last":15.045999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:49:45 2023","interfaces":[{"interface":"back","average":{"1min":18.766999999999999,"5min":18.32,"15min":19.82},"min":{"1min":11.832000000000001,"5min":11.183,"15min":7.4859999999999998},"max":{"1min":29.917000000000002,"5min":183.84700000000001,"15min":216.054},"last":15.839},{"interface":"front","average":{"1min":21.66,"5min":20.404,"15min":23.155999999999999},"min":{"1min":15.773999999999999,"5min":12.422000000000001,"15min":9.5739999999999998},"max":{"1min":34.591999999999999,"5min":186.06700000000001,"15min":199.227},"last":16.556999999999999}]},{"osd":6,"last update":"Mon Dec 11 20:49:45 2023","interfaces":[{"interface":"back","average":{"1min":11.662000000000001,"5min":13.726000000000001,"15min":18.442},"min":{"1min":6.3339999999999996,"5min":6.3339999999999996,"15min":6.3339999999999996},"max":{"1min":27.663,"5min":169.429,"15min":203.78399999999999},"last":11.818},{"interface":"front","average":{"1min":21.140000000000001,"5min":20.492999999999999,"15min":25.872},"min":{"1min":15.859999999999999,"5min":12.670999999999999,"15min":11.641999999999999},"max":{"1min":35.917999999999999,"5min":197.87299999999999,"15min":202.328},"last":17.283000000000001}]},{"osd":7,"last update":"Mon Dec 11 20:48:56 2023","interfaces":[{"interface":"back","average":{"1min":20.015000000000001,"5min":13.327,"15min":22.792999999999999},"min":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"max":{"1min":109.782,"5min":109.782,"15min":426.10700000000003},"last":10.506},{"interface":"front","average":{"1min":33.633000000000003,"5min":21.547000000000001,"15min":29.686},"min":{"1min":13.068,"5min":13.068,"15min":11.789},"max":{"1min":189.21199999999999,"5min":189.21199999999999,"15min":331.24799999999999},"last":18.16}]}]},{"osd":0,"up_from":16,"seq":68719477135,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":11.036,"15min":16.446000000000002},"min":{"1min":4.609,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":98.932000000000002,"5min":98.932000000000002,"15min":192.989},"last":10.878},{"interface":"front","average":{"1min":20.565000000000001,"5min":11.683999999999999,"15min":17.998999999999999},"min":{"1min":6.1719999999999997,"5min":5.2220000000000004,"15min":4.7300000000000004},"max":{"1min":101.66,"5min":101.66,"15min":288.91800000000001},"last":8.9469999999999992}]},{"osd":2,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":22.863,"5min":13.515000000000001,"15min":17.582999999999998},"min":{"1min":7.694,"5min":7.0620000000000003,"15min":5.2190000000000003},"max":{"1min":96.489999999999995,"5min":96.489999999999995,"15min":294.733},"last":12.382},{"interface":"front","average":{"1min":29.260999999999999,"5min":17.803999999999998,"15min":23.199999999999999},"min":{"1min":11.227,"5min":9.9109999999999996,"15min":7.8630000000000004},"max":{"1min":126.831,"5min":126.831,"15min":306.08100000000002},"last":19.486000000000001}]},{"osd":3,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.068000000000001,"5min":13.037000000000001,"15min":15.56},"min":{"1min":6.1769999999999996,"5min":5.9829999999999997,"15min":5.5979999999999999},"max":{"1min":160.999,"5min":160.999,"15min":166.49100000000001},"last":11.598000000000001},{"interface":"front","average":{"1min":29.303000000000001,"5min":17.632999999999999,"15min":20.312999999999999},"min":{"1min":10.973000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"max":{"1min":177.35400000000001,"5min":177.35400000000001,"15min":177.35400000000001},"last":13.526}]},{"osd":4,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.286000000000001,"5min":16.172999999999998,"15min":19.902999999999999},"min":{"1min":6.5970000000000004,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":169.92599999999999,"5min":169.92599999999999,"15min":169.92599999999999},"last":16.792000000000002},{"interface":"front","average":{"1min":31.260999999999999,"5min":19.547000000000001,"15min":20.800000000000001},"min":{"1min":10.371,"5min":10.371,"15min":6.6859999999999999},"max":{"1min":149.65600000000001,"5min":149.65600000000001,"15min":153.42500000000001},"last":14.512}]},{"osd":5,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":25.992999999999999,"5min":13.356,"15min":15.753},"min":{"1min":7.0940000000000003,"5min":6.5999999999999996,"15min":6.234},"max":{"1min":110.476,"5min":110.476,"15min":210.374},"last":8.5950000000000006},{"interface":"front","average":{"1min":30.498999999999999,"5min":16.635999999999999,"15min":20.227},"min":{"1min":8.7940000000000005,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":110.836,"5min":110.836,"15min":267.03199999999998},"last":16.146999999999998}]},{"osd":6,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":24.010999999999999,"5min":14.885,"15min":18.350999999999999},"min":{"1min":9.0850000000000009,"5min":8.0050000000000008,"15min":6.4790000000000001},"max":{"1min":111.423,"5min":111.423,"15min":161.09899999999999},"last":9.6530000000000005},{"interface":"front","average":{"1min":33.442999999999998,"5min":19.559000000000001,"15min":23.120000000000001},"min":{"1min":11.138999999999999,"5min":10.305999999999999,"15min":9.4440000000000008},"max":{"1min":186.089,"5min":186.089,"15min":246.738},"last":17.718}]},{"osd":7,"last update":"Mon Dec 11 20:48:57 2023","interfaces":[{"interface":"back","average":{"1min":31.001000000000001,"5min":19.183,"15min":21.738},"min":{"1min":9.4049999999999994,"5min":9.4049999999999994,"15min":8.8789999999999996},"max":{"1min":187.00299999999999,"5min":187.00299999999999,"15min":187.00299999999999},"last":18.341000000000001},{"interface":"front","average":{"1min":29.199999999999999,"5min":19.706,"15min":23.911999999999999},"min":{"1min":12.224,"5min":11.782999999999999,"15min":7.7640000000000002},"max":{"1min":127.563,"5min":127.563,"15min":264.45100000000002},"last":20.405999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542539,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65168,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306672,"statfs":{"total":96636764160,"available":96570032128,"internally_reserved":0,"allocated":2105344,"data_stored":556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":13.959,"5min":10.147,"15min":15.048999999999999},"min":{"1min":4.5049999999999999,"5min":4.4409999999999998,"15min":4.1879999999999997},"max":{"1min":40.877000000000002,"5min":40.877000000000002,"15min":248.19900000000001},"last":8.2569999999999997},{"interface":"front","average":{"1min":17.295999999999999,"5min":10.696999999999999,"15min":13.792},"min":{"1min":5.1589999999999998,"5min":5.1059999999999999,"15min":4.1449999999999996},"max":{"1min":77.028000000000006,"5min":77.028000000000006,"15min":169.541},"last":13.943}]},{"osd":1,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":23.628,"5min":14.050000000000001,"15min":18.832000000000001},"min":{"1min":7.1269999999999998,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":116.298,"5min":116.298,"15min":310.53500000000003},"last":9.3719999999999999},{"interface":"front","average":{"1min":21.18,"5min":13.920999999999999,"15min":18.367999999999999},"min":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":4.7380000000000004},"max":{"1min":109.753,"5min":109.753,"15min":269.65100000000001},"last":15.741}]},{"osd":2,"last update":"Mon Dec 11 20:49:17 2023","interfaces":[{"interface":"back","average":{"1min":15.798999999999999,"5min":14.196999999999999,"15min":18.044},"min":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":6.1429999999999998},"max":{"1min":50.573,"5min":50.573,"15min":255.31700000000001},"last":12.428000000000001},{"interface":"front","average":{"1min":13.231999999999999,"5min":11.869,"15min":17.018000000000001},"min":{"1min":6.9139999999999997,"5min":6.6310000000000002,"15min":5.4199999999999999},"max":{"1min":40.917999999999999,"5min":40.917999999999999,"15min":291.67899999999997},"last":14.904999999999999}]},{"osd":3,"last update":"Mon Dec 11 20:49:10 2023","interfaces":[{"interface":"back","average":{"1min":21.579000000000001,"5min":14.327999999999999,"15min":18.846},"min":{"1min":8.2970000000000006,"5min":7.6790000000000003,"15min":7.2460000000000004},"max":{"1min":132.065,"5min":132.065,"15min":260.55099999999999},"last":16.559000000000001},{"interface":"front","average":{"1min":19.283999999999999,"5min":16.18,"15min":21.841000000000001},"min":{"1min":7.0860000000000003,"5min":7.0860000000000003,"15min":7.0030000000000001},"max":{"1min":50.997999999999998,"5min":50.997999999999998,"15min":320.53899999999999},"last":17.190999999999999}]},{"osd":5,"last update":"Mon Dec 11 20:49:00 2023","interfaces":[{"interface":"back","average":{"1min":26.222000000000001,"5min":18.422999999999998,"15min":21.838999999999999},"min":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":7.7359999999999998},"max":{"1min":132.37799999999999,"5min":132.37799999999999,"15min":298.69499999999999},"last":10.086},{"interface":"front","average":{"1min":22.783999999999999,"5min":15.782,"15min":21.024999999999999},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":81.689999999999998,"5min":81.689999999999998,"15min":309.88200000000001},"last":18.661999999999999}]},{"osd":6,"last update":"Mon Dec 11 20:49:10 2023","interfaces":[{"interface":"back","average":{"1min":25.664999999999999,"5min":17.158000000000001,"15min":19.324999999999999},"min":{"1min":10.003,"5min":8.6419999999999995,"15min":5.4459999999999997},"max":{"1min":128.589,"5min":128.589,"15min":290.57600000000002},"last":21.379999999999999},{"interface":"front","average":{"1min":24.858000000000001,"5min":15.848000000000001,"15min":19.082999999999998},"min":{"1min":6.5469999999999997,"5min":6.5469999999999997,"15min":6.5389999999999997},"max":{"1min":107.01600000000001,"5min":107.01600000000001,"15min":285.52499999999998},"last":19.294}]},{"osd":7,"last update":"Mon Dec 11 20:49:48 2023","interfaces":[{"interface":"back","average":{"1min":20.119,"5min":18.478000000000002,"15min":20.548999999999999},"min":{"1min":12.824999999999999,"5min":10.587,"15min":7.3940000000000001},"max":{"1min":40.039000000000001,"5min":69.731999999999999,"15min":297.63299999999998},"last":22.222999999999999},{"interface":"front","average":{"1min":18.094000000000001,"5min":17.175999999999998,"15min":21.981000000000002},"min":{"1min":8.7270000000000003,"5min":7.3639999999999999,"15min":7.3639999999999999},"max":{"1min":47.399000000000001,"5min":98.728999999999999,"15min":364.18799999999999},"last":19.878}]}]},{"osd":7,"up_from":13,"seq":55834575246,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65164,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2101248,"data_stored":556181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:49:20 2023","interfaces":[{"interface":"back","average":{"1min":12.233000000000001,"5min":10.454000000000001,"15min":15.913},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":3.8969999999999998},"max":{"1min":41.088000000000001,"5min":41.088000000000001,"15min":280.62200000000001},"last":8.8529999999999998},{"interface":"front","average":{"1min":12.196,"5min":9.984,"15min":16.021000000000001},"min":{"1min":5.516,"5min":5.516,"15min":4.9800000000000004},"max":{"1min":49.896000000000001,"5min":49.896000000000001,"15min":294.44299999999998},"last":7.3540000000000001}]},{"osd":1,"last update":"Mon Dec 11 20:49:09 2023","interfaces":[{"interface":"back","average":{"1min":12.32,"5min":11.670999999999999,"15min":16.097999999999999},"min":{"1min":6.5209999999999999,"5min":5.5519999999999996,"15min":5.4489999999999998},"max":{"1min":21.082000000000001,"5min":21.082000000000001,"15min":217.40100000000001},"last":14.304},{"interface":"front","average":{"1min":13.526999999999999,"5min":12.893000000000001,"15min":17.603000000000002},"min":{"1min":9.3949999999999996,"5min":6.3730000000000002,"15min":6.3730000000000002},"max":{"1min":22.927,"5min":22.927,"15min":247.75299999999999},"last":17.030000000000001}]},{"osd":2,"last update":"Mon Dec 11 20:49:09 2023","interfaces":[{"interface":"back","average":{"1min":20.245999999999999,"5min":14.756,"15min":18.093},"min":{"1min":9.5399999999999991,"5min":7.7839999999999998,"15min":7.1920000000000002},"max":{"1min":143.51599999999999,"5min":143.51599999999999,"15min":168.923},"last":9.6210000000000004},{"interface":"front","average":{"1min":37.488,"5min":20.488,"15min":24.061},"min":{"1min":11.813000000000001,"5min":11.714,"15min":11.016},"max":{"1min":297.77999999999997,"5min":297.77999999999997,"15min":297.77999999999997},"last":11.456}]},{"osd":3,"last update":"Mon Dec 11 20:49:09 2023","interfaces":[{"interface":"back","average":{"1min":13.353,"5min":9.3569999999999993,"15min":12.16},"min":{"1min":6.4029999999999996,"5min":5.1879999999999997,"15min":5.1879999999999997},"max":{"1min":86.045000000000002,"5min":86.045000000000002,"15min":199.89500000000001},"last":12.167999999999999},{"interface":"front","average":{"1min":23.440999999999999,"5min":17.196000000000002,"15min":20.536999999999999},"min":{"1min":11.305,"5min":9.6389999999999993,"15min":9.2240000000000002},"max":{"1min":143.214,"5min":143.214,"15min":210.63200000000001},"last":10.673}]},{"osd":4,"last update":"Mon Dec 11 20:49:47 2023","interfaces":[{"interface":"back","average":{"1min":13.548999999999999,"5min":16.082999999999998,"15min":15.526999999999999},"min":{"1min":9.2620000000000005,"5min":6.9279999999999999,"15min":6.1900000000000004},"max":{"1min":19.629000000000001,"5min":166.67099999999999,"15min":166.67099999999999},"last":13.000999999999999},{"interface":"front","average":{"1min":9.9420000000000002,"5min":12.705,"15min":16.677},"min":{"1min":6.8579999999999997,"5min":6.2510000000000003,"15min":6.2510000000000003},"max":{"1min":17.626000000000001,"5min":124.88,"15min":203.36600000000001},"last":15.077}]},{"osd":5,"last update":"Mon Dec 11 20:49:47 2023","interfaces":[{"interface":"back","average":{"1min":17.952000000000002,"5min":21.808,"15min":25.408000000000001},"min":{"1min":11.843999999999999,"5min":10.476000000000001,"15min":9.7810000000000006},"max":{"1min":24.739999999999998,"5min":206.904,"15min":268.98000000000002},"last":17.698},{"interface":"front","average":{"1min":14.510999999999999,"5min":16.725999999999999,"15min":18.757999999999999},"min":{"1min":10.013999999999999,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":18.785,"5min":127.8,"15min":283.61900000000003},"last":15.891}]},{"osd":6,"last update":"Mon Dec 11 20:49:49 2023","interfaces":[{"interface":"back","average":{"1min":15.622,"5min":16.751000000000001,"15min":20.338000000000001},"min":{"1min":10.962999999999999,"5min":8.8339999999999996,"15min":6.6479999999999997},"max":{"1min":21.986000000000001,"5min":115.38800000000001,"15min":282.71199999999999},"last":19.657},{"interface":"front","average":{"1min":18.632000000000001,"5min":19.742000000000001,"15min":24.126000000000001},"min":{"1min":12.372999999999999,"5min":10.999000000000001,"15min":9.7569999999999997},"max":{"1min":25.012,"5min":126.97199999999999,"15min":260.13400000000001},"last":18.387}]}]},{"osd":5,"up_from":13,"seq":55834575244,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69716,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302124,"statfs":{"total":96636764160,"available":96565374976,"internally_reserved":0,"allocated":2502656,"data_stored":954021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":13.065,"5min":10.132999999999999,"15min":13.657999999999999},"min":{"1min":4.4960000000000004,"5min":4.4509999999999996,"15min":4.3849999999999998},"max":{"1min":52.319000000000003,"5min":52.319000000000003,"15min":121.80500000000001},"last":13.353999999999999},{"interface":"front","average":{"1min":13.058999999999999,"5min":9.7390000000000008,"15min":16.152999999999999},"min":{"1min":6.3780000000000001,"5min":4.5369999999999999,"15min":4.4349999999999996},"max":{"1min":49.723999999999997,"5min":49.723999999999997,"15min":307.66000000000003},"last":6.0549999999999997}]},{"osd":1,"last update":"Mon Dec 11 20:49:45 2023","interfaces":[{"interface":"back","average":{"1min":19.277999999999999,"5min":11.728999999999999,"15min":13.565},"min":{"1min":5.516,"5min":5.3449999999999998,"15min":5.0019999999999998},"max":{"1min":118.562,"5min":118.562,"15min":131.70500000000001},"last":6.7210000000000001},{"interface":"front","average":{"1min":16.861000000000001,"5min":12.818,"15min":16.437999999999999},"min":{"1min":8.2119999999999997,"5min":6.0609999999999999,"15min":5.7169999999999996},"max":{"1min":71.447000000000003,"5min":71.447000000000003,"15min":202.07300000000001},"last":10.074}]},{"osd":2,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.559000000000001,"5min":15.26,"15min":18.596},"min":{"1min":8.6720000000000006,"5min":7.7460000000000004,"15min":6.9470000000000001},"max":{"1min":109.023,"5min":109.023,"15min":203.67699999999999},"last":15.029},{"interface":"front","average":{"1min":23.201000000000001,"5min":15.288,"15min":19.117000000000001},"min":{"1min":9.0129999999999999,"5min":7.8760000000000003,"15min":6.8719999999999999},"max":{"1min":93.819999999999993,"5min":93.819999999999993,"15min":211.67500000000001},"last":11.939}]},{"osd":3,"last update":"Mon Dec 11 20:49:45 2023","interfaces":[{"interface":"back","average":{"1min":14.701000000000001,"5min":15.079000000000001,"15min":18.556999999999999},"min":{"1min":8.4299999999999997,"5min":8.0700000000000003,"15min":7.7590000000000003},"max":{"1min":23.372,"5min":58.795000000000002,"15min":141.10900000000001},"last":14.252000000000001},{"interface":"front","average":{"1min":14.83,"5min":15.287000000000001,"15min":19.271999999999998},"min":{"1min":7.6909999999999998,"5min":7.6909999999999998,"15min":7.6769999999999996},"max":{"1min":23.428999999999998,"5min":68.209999999999994,"15min":207.58799999999999},"last":17.393000000000001}]},{"osd":4,"last update":"Mon Dec 11 20:48:59 2023","interfaces":[{"interface":"back","average":{"1min":21.547000000000001,"5min":16.661000000000001,"15min":19.132000000000001},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":6.8360000000000003},"max":{"1min":69.195999999999998,"5min":69.195999999999998,"15min":132.64500000000001},"last":16.145},{"interface":"front","average":{"1min":33.280999999999999,"5min":20.861000000000001,"15min":23.035},"min":{"1min":14.875999999999999,"5min":9.9390000000000001,"15min":8.2590000000000003},"max":{"1min":199.828,"5min":199.828,"15min":199.828},"last":19.792000000000002}]},{"osd":6,"last update":"Mon Dec 11 20:49:45 2023","interfaces":[{"interface":"back","average":{"1min":15.705,"5min":13.996,"15min":15.800000000000001},"min":{"1min":6.2510000000000003,"5min":6.234,"15min":5.8789999999999996},"max":{"1min":119.57299999999999,"5min":199.03800000000001,"15min":199.03800000000001},"last":11.342000000000001},{"interface":"front","average":{"1min":22.210000000000001,"5min":20.329000000000001,"15min":19.279},"min":{"1min":12.973000000000001,"5min":11.548999999999999,"15min":6.1479999999999997},"max":{"1min":111.997,"5min":113.911,"15min":113.911},"last":18.893999999999998}]},{"osd":7,"last update":"Mon Dec 11 20:49:45 2023","interfaces":[{"interface":"back","average":{"1min":18.776,"5min":19.68,"15min":21.314},"min":{"1min":15.532,"5min":10.284000000000001,"15min":8.3239999999999998},"max":{"1min":28.050000000000001,"5min":63.686999999999998,"15min":106.499},"last":20.617000000000001},{"interface":"front","average":{"1min":14.859,"5min":15.664,"15min":18.968},"min":{"1min":7.7069999999999999,"5min":7.7069999999999999,"15min":7.7069999999999999},"max":{"1min":21.945,"5min":56.078000000000003,"15min":141.209},"last":16.782}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-11T20:49:54.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-11T20:49:54.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.338+0000 7fd5161dc640 10 monclient: get_monmap_and_config 2023-12-11T20:49:54.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.338+0000 7fd5161dc640 10 monclient: build_initial_monmap 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.338+0000 7fd5161dc640 10 monclient: monmap: 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:49:54.339641+0000 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:49:54.339641+0000 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c-legacy 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.133:3300/0 mon.noname-c 2023-12-11T20:49:54.338 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.338+0000 7fd5161dc640 1 Processor -- start 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 1 -- start start 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient: init 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5100a8990 addr v1:172.21.15.86:6789/0 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5101427b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient(hunting): picked mon.noname-b con 0x7fd5100a8d70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd51013ef10 addr v1:172.21.15.133:6789/0 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd510142cf0 con 0x7fd5100a8990 2023-12-11T20:49:54.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5100aabe0 con 0x7fd51013ef10 2023-12-11T20:49:54.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:54.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5100aaec0 con 0x7fd5100a8d70 2023-12-11T20:49:54.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:54.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.339+0000 7fd5161dc640 10 monclient(hunting): authenticate will time out at 3480.669885s 2023-12-11T20:49:54.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.340+0000 7fd50f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd5100a8990 0x7fd5100b21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41528/0 (socket says 172.21.15.86:41528) 2023-12-11T20:49:54.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.340+0000 7fd50f7fe640 1 -- 172.21.15.86:0/3293186118 learned_addr learned my addr 172.21.15.86:0/3293186118 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-12-11T20:49:54.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.341+0000 7fd50effd640 1 --2- 172.21.15.86:0/3293186118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5101427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:54.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.341+0000 7fd50effd640 10 monclient(hunting): get_auth_request con 0x7fd5100a8d70 auth_method 0 2023-12-11T20:49:54.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.341+0000 7fd50effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:54.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.341+0000 7fd50effd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:54.342 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.341+0000 7fd50effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50e7fc640 1 -- 172.21.15.86:0/3293186118 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619919650 0 0) 0x7fd510142cf0 con 0x7fd5100a8990 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50e7fc640 10 monclient(hunting): _negotiate 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50e7fc640 10 monclient(hunting): my global_id is 16974 2023-12-11T20:49:54.343 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.342+0000 7fd50e7fc640 1 -- 172.21.15.86:0/3293186118 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f8003520 con 0x7fd5100a8990 2023-12-11T20:49:54.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.343+0000 7fd50effd640 10 monclient(hunting): handle_auth_done global_id 19091 payload 274 2023-12-11T20:49:54.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.343+0000 7fd50effd640 1 -- 172.21.15.86:0/3293186118 >> v1:172.21.15.133:6789/0 conn(0x7fd51013ef10 legacy=0x7fd51013f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-11T20:49:54.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.343+0000 7fd50effd640 1 -- 172.21.15.86:0/3293186118 >> v1:172.21.15.86:6789/0 conn(0x7fd5100a8990 legacy=0x7fd5100b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:54.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.343+0000 7fd50effd640 10 monclient: _finish_hunting 0 2023-12-11T20:49:54.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.343+0000 7fd50effd640 1 monclient: found mon.noname-b 2023-12-11T20:49:54.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.343+0000 7fd50effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:54.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.343+0000 7fd50effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2023-12-11T20:49:54.344 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.343+0000 7fd50effd640 1 -- 172.21.15.86:0/3293186118 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd510142260 con 0x7fd5100a8d70 2023-12-11T20:49:54.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.345+0000 7fd50effd640 1 --2- 172.21.15.86:0/3293186118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5101427b0 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fd5040097d0 tx=0x7fd50402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a76650f3bf51f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 1 -- 172.21.15.86:0/3293186118 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504031800 con 0x7fd5100a8d70 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: dump: 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:54.349 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: _finish_auth 0 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: _check_auth_tickets 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 1 -- 172.21.15.86:0/3293186118 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd504031960 con 0x7fd5100a8d70 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd5161dc640 5 monclient: authenticate success, global_id 19091 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd5161dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 1 -- 172.21.15.86:0/3293186118 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504031cf0 con 0x7fd5100a8d70 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd5161dc640 10 monclient: get_monmap_and_config success 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd5161dc640 4 set_mon_vals no callback set 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.349+0000 7fd50e7fc640 10 monclient: dump: 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:54.352 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:54.352 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:54.352 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 10 monclient: shutdown 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 1 -- 172.21.15.86:0/3293186118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 msgr2=0x7fd5101427b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 1 --2- 172.21.15.86:0/3293186118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5101427b0 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fd5040097d0 tx=0x7fd50402b8a0 comp rx=0 tx=0).stop 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 1 -- 172.21.15.86:0/3293186118 shutdown_connections 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 1 --2- 172.21.15.86:0/3293186118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5101427b0 unknown :-1 s=CLOSED pgs=3348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 1 -- 172.21.15.86:0/3293186118 >> 172.21.15.86:0/3293186118 conn(0x7fd5100b8310 msgr2=0x7fd5100b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 1 -- 172.21.15.86:0/3293186118 shutdown_connections 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 1 -- 172.21.15.86:0/3293186118 wait complete. 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 10 monclient: build_initial_monmap 2023-12-11T20:49:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.350+0000 7fd5161dc640 10 monclient: monmap: 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.noname-b 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.351+0000 7fd5161dc640 1 librados: starting msgr at 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.351+0000 7fd5161dc640 1 librados: starting objecter 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.351+0000 7fd5161dc640 1 Processor -- start 2023-12-11T20:49:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.351+0000 7fd5161dc640 1 -- start start 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.351+0000 7fd5161dc640 1 librados: setting wanted keys 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.351+0000 7fd5161dc640 1 librados: calling monclient init 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.351+0000 7fd5161dc640 10 monclient: init 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.351+0000 7fd5161dc640 10 monclient: _reopen_session rank -1 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5100a8990 0x7fd5100ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient(hunting): picked mon.noname-a con 0x7fd5100a8990 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5100ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient(hunting): picked mon.noname-c con 0x7fd5100a8d70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd51013ef10 0x7fd5101da8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient(hunting): picked mon.noname-b con 0x7fd51013ef10 addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5100ac910 con 0x7fd5100a8990 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5100ae6a0 con 0x7fd51013ef10 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient(hunting): start opening mon connection 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5100ad780 con 0x7fd5100a8d70 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient(hunting): _renew_subs 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd5161dc640 10 monclient(hunting): authenticate will time out at 3480.682198s 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd50ffff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd51013ef10 0x7fd5101da8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd50ffff640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd51013ef10 0x7fd5101da8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.86:60598/0 (socket says 172.21.15.86:60598) 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd50ffff640 1 -- 172.21.15.86:0/967148165 learned_addr learned my addr 172.21.15.86:0/967148165 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd50ffff640 10 monclient(hunting): get_auth_request con 0x7fd51013ef10 auth_method 0 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd50ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd50ffff640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.352+0000 7fd50ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.353+0000 7fd50effd640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5100ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.353+0000 7fd50ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.353+0000 7fd50ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.353+0000 7fd50ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.353+0000 7fd50effd640 10 monclient(hunting): get_auth_request con 0x7fd5100a8d70 auth_method 0 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.353+0000 7fd50effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.353+0000 7fd50effd640 10 monclient(hunting): _init_auth method 2 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.353+0000 7fd50effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.354+0000 7fd50effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.354+0000 7fd50effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-11T20:49:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.354+0000 7fd50effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-11T20:49:54.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 10 monclient(hunting): handle_auth_done global_id 21130 payload 824 2023-12-11T20:49:54.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 1 -- 172.21.15.86:0/967148165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 msgr2=0x7fd5100ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5100ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 1 -- 172.21.15.86:0/967148165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5100a8990 msgr2=0x7fd5100ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5100a8990 0x7fd5100ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 10 monclient: _finish_hunting 0 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 1 monclient: found mon.noname-b 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.133:3300/0 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.356+0000 7fd50ffff640 1 -- 172.21.15.86:0/967148165 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd504009400 con 0x7fd51013ef10 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.358+0000 7fd50ffff640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd51013ef10 0x7fd5101da8b0 secure :-1 s=READY pgs=5137 cs=0 l=1 rev1=1 crypto rx=0x7fd50800b990 tx=0x7fd50800be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.358+0000 7fd4f7fff640 10 client.?.objecter ms_handle_connect 0x7fd51013ef10 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.358+0000 7fd4f7fff640 10 client.?.objecter resend_mon_ops 2023-12-11T20:49:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd4f7fff640 1 -- 172.21.15.86:0/967148165 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50800cba0 con 0x7fd51013ef10 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd4f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd4f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd4f7fff640 10 monclient: dump: 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:54.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd4f7fff640 10 monclient: _finish_auth 0 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd4f7fff640 10 monclient: _check_auth_tickets 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd4f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 5 monclient: authenticate success, global_id 21130 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 10 monclient: _renew_subs 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5100ada60 con 0x7fd51013ef10 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 10 client.21130.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 10 monclient: _renew_subs 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5100ab670 con 0x7fd51013ef10 2023-12-11T20:49:54.363 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.361+0000 7fd5161dc640 1 librados: init done 2023-12-11T20:49:54.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.362+0000 7fd5161dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:54.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.362+0000 7fd5161dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:54.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.362+0000 7fd5161dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:54.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.362+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5100ac750 con 0x7fd51013ef10 2023-12-11T20:49:54.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.363+0000 7fd4f7fff640 1 -- 172.21.15.86:0/967148165 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5080098d0 con 0x7fd51013ef10 2023-12-11T20:49:54.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.363+0000 7fd4f7fff640 10 monclient: handle_config config(0 keys) v1 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.363+0000 7fd4f7fff640 1 -- 172.21.15.86:0/967148165 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50801e440 con 0x7fd51013ef10 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.363+0000 7fd4f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.363+0000 7fd4f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.363+0000 7fd4f7fff640 10 monclient: dump: 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid bfc699ed-5134-41ce-a2e4-de833b93556d 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 2023-12-11T20:15:13.894666+0000 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 18 (reef) 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.b 2023-12-11T20:49:54.365 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-12-11T20:49:54.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:49:54.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.363+0000 7fd50dffb640 4 set_mon_vals no callback set 2023-12-11T20:49:54.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.602+0000 7fd4f7fff640 1 -- 172.21.15.86:0/967148165 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd508014440 con 0x7fd51013ef10 2023-12-11T20:49:54.602 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.603+0000 7fd4f7fff640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4fc06c850 0x7fd4fc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-11T20:49:54.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.603+0000 7fd50f7fe640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4fc06c850 0x7fd4fc06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-11T20:49:54.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.603+0000 7fd50f7fe640 10 monclient: get_auth_request con 0x7fd4fc06c850 auth_method 0 2023-12-11T20:49:54.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.603+0000 7fd50f7fe640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4fc06c850 0x7fd4fc06ed10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd500000f60 tx=0x7fd5000078f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-11T20:49:54.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.603+0000 7fd4f7fff640 10 client.21130.objecter ms_handle_connect 0x7fd4fc06c850 2023-12-11T20:49:54.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.607+0000 7fd4f7fff640 1 -- 172.21.15.86:0/967148165 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(1057..1057 src has 1000..1057) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fd508058ab0 con 0x7fd51013ef10 2023-12-11T20:49:54.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.607+0000 7fd4f7fff640 10 client.21130.objecter ms_dispatch 0x7fd510000b90 osd_map(1057..1057 src has 1000..1057) v4 2023-12-11T20:49:54.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.607+0000 7fd4f7fff640 3 client.21130.objecter handle_osd_map got epochs [1057,1057] > 0 2023-12-11T20:49:54.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.607+0000 7fd4f7fff640 3 client.21130.objecter handle_osd_map decoding full epoch 1057 2023-12-11T20:49:54.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.607+0000 7fd4f7fff640 20 client.21130.objecter dump_active .. 0 homeless 2023-12-11T20:49:54.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.617+0000 7fd4f7fff640 1 -- 172.21.15.86:0/967148165 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd508054df0 con 0x7fd51013ef10 2023-12-11T20:49:54.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.617+0000 7fd4f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-11T20:49:54.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.617+0000 7fd4f7fff640 10 monclient: _finish_command 1 = system:0 2023-12-11T20:49:54.794 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.794+0000 7fd5161dc640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-11T20:49:54.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.794+0000 7fd5161dc640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-11T20:49:54.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.794+0000 7fd5161dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.133:3300/0 2023-12-11T20:49:54.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:54.794+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fd51004e840 con 0x7fd51013ef10 2023-12-11T20:49:54.819 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:07.927 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:54.822 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:07.930 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:54.836 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:07.944 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:54.874 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:07.942 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:54.875 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:07.944 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:54.888 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:07.996 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:54.889 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:07.957 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:54.891 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:07.999 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:54.906 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:08.014 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.347+0000 7fd4f7fff640 1 -- 172.21.15.86:0/967148165 <== mon.1 v2:172.21.15.133:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd50805cb50 con 0x7fd51013ef10 2023-12-11T20:49:55.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.347+0000 7fd4f7fff640 10 monclient: handle_config config(1 keys) v1 2023-12-11T20:49:55.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.347+0000 7fd50dffb640 4 set_mon_vals no callback set 2023-12-11T20:49:55.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.347+0000 7fd50dffb640 10 set_mon_vals mon_health_to_clog = false 2023-12-11T20:49:55.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.347+0000 7fd4f7fff640 1 -- 172.21.15.86:0/967148165 <== mon.1 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fd5080230c0 con 0x7fd51013ef10 2023-12-11T20:49:55.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.347+0000 7fd4f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-11T20:49:55.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.347+0000 7fd4f7fff640 10 monclient: _finish_command 2 = system:0 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 10 librados: watch_flush enter 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 10 librados: watch_flush exit 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 20 client.21130.objecter shutdown clearing up homeless session... 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 10 client.21130.objecter successfully canceled tick 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4fc06c850 msgr2=0x7fd4fc06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4fc06c850 0x7fd4fc06ed10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd500000f60 tx=0x7fd5000078f0 comp rx=0 tx=0).stop 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 10 monclient: shutdown 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd51013ef10 msgr2=0x7fd5101da8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd51013ef10 0x7fd5101da8b0 secure :-1 s=READY pgs=5137 cs=0 l=1 rev1=1 crypto rx=0x7fd50800b990 tx=0x7fd50800be50 comp rx=0 tx=0).stop 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 shutdown_connections 2023-12-11T20:49:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:6800/1322765564,v1:172.21.15.133:6801/1322765564] conn(0x7fd4fc06c850 0x7fd4fc06ed10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:55.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5100a8d70 0x7fd5100ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:55.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd51013ef10 0x7fd5101da8b0 unknown :-1 s=CLOSED pgs=5137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:55.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.349+0000 7fd5161dc640 1 --2- 172.21.15.86:0/967148165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5100a8990 0x7fd5100ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-11T20:49:55.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.350+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 >> 172.21.15.86:0/967148165 conn(0x7fd5100b8310 msgr2=0x7fd51013bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-11T20:49:55.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.350+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 shutdown_connections 2023-12-11T20:49:55.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.350+0000 7fd5161dc640 1 -- 172.21.15.86:0/967148165 wait complete. 2023-12-11T20:49:55.351 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-11T20:49:55.350+0000 7fd5161dc640 1 librados: shutdown 2023-12-11T20:49:55.363 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-11T20:49:55.363 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-11T20:49:55.363 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-11T20:49:55.363 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:49:55.376 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:49:55.367+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86819) UID: 0 2023-12-11T20:49:55.380 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-12-11T20:49:55.369+0000 b47e640 -1 osd.0 1057 *** Got signal Terminated *** 2023-12-11T20:49:55.580 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:08.648 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.581 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:08.649 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.588 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:08.657 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.619 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:08.727 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.620 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:08.728 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.628 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:08.736 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.665 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:08.773 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.666 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:08.774 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:49:55.674 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:08.782 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.814 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:22.921 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.858 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:22.926 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.860 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:22.928 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.862 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:22.970 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.878 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:22.946 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.884 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:22.992 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.927 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:23.035 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.930 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:23.038 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:09.949 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:23.057 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.873 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:27.981 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.875 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:27.983 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.887 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:27.995 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.924 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:27.992 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.940 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:28.008 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.943 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:28.051 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.948 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:28.056 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.952 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:28.020 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:14.961 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:28.069 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.905 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:43.012 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.909 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:43.017 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.924 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:34:43.032 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.956 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:43.064 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.963 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:43.069 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.963 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:43.030 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.967 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:43.035 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.975 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:34:43.083 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:29.980 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:34:43.048 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:49.954 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:03.061 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:49.957 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:03.065 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:49.981 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:03.089 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.005 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:03.072 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.007 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:03.075 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.011 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:03.119 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.014 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:03.122 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.022 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:03.090 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.037 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:03.145 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.611 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:03.679 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.611 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:03.680 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.623 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:03.691 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.631 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:03.739 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.632 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:03.740 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.644 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:03.752 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.654 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:03.762 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.655 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:03.763 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:50.666 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:03.774 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:54.968 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:08.076 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:54.972 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:08.080 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:54.983 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:08.091 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:55.018 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:08.125 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:55.019 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:08.087 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:55.020 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:08.128 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:55.021 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:08.089 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:55.030 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:08.138 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:55.031 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:08.099 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:50:55.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-11T20:50:56.379 INFO:tasks.ceph.osd.0:Stopped 2023-12-11T20:50:56.380 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-11T20:50:56.380 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:50:56.393 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:50:56.386+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86823) UID: 0 2023-12-11T20:50:56.394 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-12-11T20:50:56.388+0000 b47e640 -1 osd.1 1062 *** Got signal Terminated *** 2023-12-11T20:51:01.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-12-11T20:51:10.012 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:23.120 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:10.016 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:23.124 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:10.030 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:23.138 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:10.063 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:23.170 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:10.064 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:23.172 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:10.065 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:23.133 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:10.066 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:23.134 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:10.080 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:23.188 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:10.082 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:23.150 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.034 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:28.141 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.035 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:28.143 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.047 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:28.155 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.081 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:28.189 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.084 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:28.192 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.088 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:28.156 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.090 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:28.158 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.095 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:28.203 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:15.101 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:28.169 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.066 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:43.173 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.068 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:43.176 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.090 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:43.198 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.118 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:43.186 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.122 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:43.190 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.127 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:43.235 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.130 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:43.238 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.139 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:43.207 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:30.154 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:43.259 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.129 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:48.237 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.133 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:48.241 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.176 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:48.244 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.179 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:48.286 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.182 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:48.250 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.182 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:48.291 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.184 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:35:48.292 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.219 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:35:48.287 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:35.223 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:35:48.331 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.138 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:03.246 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.140 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:03.248 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.169 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:03.276 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.193 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:03.261 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.195 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:03.263 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.195 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:03.303 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.198 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:03.306 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.214 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:03.282 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:50.225 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:03.333 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.152 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:08.260 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.156 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:08.264 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.172 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:08.280 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.201 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:08.309 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.203 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:08.311 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.205 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:08.273 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.208 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:08.276 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.217 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:08.325 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:51:55.218 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:08.286 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.191 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:28.298 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.240 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:28.348 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.263 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:28.371 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.285 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:28.393 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.289 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:28.357 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.290 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:28.398 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.294 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:28.362 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.310 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:28.378 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:15.312 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:28.420 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:27.898 INFO:tasks.ceph.osd.1:Stopped 2023-12-11T20:52:27.898 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-11T20:52:27.898 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:52:27.910 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:52:27.903+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86827) UID: 0 2023-12-11T20:52:27.913 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-12-11T20:52:27.904+0000 b47e640 -1 osd.2 1064 *** Got signal Terminated *** 2023-12-11T20:52:28.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-11T20:52:30.346 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:43.454 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:30.349 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:43.456 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:30.365 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:43.473 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:30.398 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:43.466 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:30.399 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:43.467 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:30.413 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:43.521 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:30.415 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:43.483 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:30.415 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:43.523 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:30.432 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:43.540 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:33.999 INFO:tasks.ceph.osd.2:Stopped 2023-12-11T20:52:34.000 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-11T20:52:34.000 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:52:34.012 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:52:34.006+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86837) UID: 0 2023-12-11T20:52:34.012 INFO:tasks.ceph.osd.3.smithi086.stderr:2023-12-11T20:52:34.007+0000 b47e640 -1 osd.3 1067 *** Got signal Terminated *** 2023-12-11T20:52:34.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-11T20:52:34.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-11T20:52:35.387 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:48.495 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:35.389 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:48.497 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:35.400 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:48.508 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:35.436 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:48.544 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:35.438 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:48.506 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:35.438 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:48.546 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:35.439 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:48.507 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:35.449 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:48.517 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:35.452 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:48.560 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:40.102 INFO:tasks.ceph.osd.3:Stopped 2023-12-11T20:52:40.102 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-11T20:52:40.102 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:52:40.115 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:52:40.108+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86777) UID: 0 2023-12-11T20:52:40.115 INFO:tasks.ceph.osd.4.smithi133.stderr:2023-12-11T20:52:40.109+0000 b47e640 -1 osd.4 1070 *** Got signal Terminated *** 2023-12-11T20:52:40.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-12-11T20:52:40.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-11T20:52:45.405 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:58.512 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:45.451 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:58.559 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:45.452 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:58.519 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:45.459 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:58.527 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:45.473 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:36:58.541 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:45.487 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:36:58.595 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:45.501 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:58.609 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:45.502 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:58.610 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:45.525 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:36:58.633 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:46.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-12-11T20:52:46.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-12-11T20:52:46.204 INFO:tasks.ceph.osd.4:Stopped 2023-12-11T20:52:46.204 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-11T20:52:46.204 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:52:46.216 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:52:46.209+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86779) UID: 0 2023-12-11T20:52:46.218 INFO:tasks.ceph.osd.5.smithi133.stderr:2023-12-11T20:52:46.211+0000 b47e640 -1 osd.5 1073 *** Got signal Terminated *** 2023-12-11T20:52:51.883 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-12-11T20:52:55.317 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:08.425 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:55.362 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:08.430 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:55.363 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:08.431 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:55.365 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:08.473 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:55.378 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:08.446 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:55.386 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:08.494 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:55.431 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:08.539 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:55.433 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:08.541 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:52:55.450 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:08.558 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.387 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:18.495 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.389 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:18.497 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.406 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:18.514 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.432 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:18.540 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.438 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:18.546 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.439 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:18.507 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.441 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:18.509 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.452 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:18.559 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:05.452 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:18.520 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.446 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:28.554 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.450 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:28.558 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.462 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:28.570 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.493 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:28.601 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.495 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:28.603 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.500 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:28.568 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.501 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:28.570 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.510 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:28.618 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:15.514 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:28.582 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.450 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:38.557 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.453 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:38.561 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.465 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:38.573 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.499 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:38.567 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.500 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:38.568 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.504 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:38.612 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.506 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:38.614 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.510 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:38.578 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:25.516 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:38.624 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.492 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:53.600 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.494 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:53.602 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.508 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:53.616 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.538 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:53.606 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.539 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:53.607 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.555 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:53.623 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.560 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:53.668 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.562 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:53.670 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:40.579 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:53.687 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.500 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:58.607 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.501 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:58.609 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.516 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:37:58.624 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.543 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:58.611 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.546 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:58.615 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.558 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:37:58.626 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.562 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:58.670 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.570 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:58.677 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:45.582 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:37:58.690 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.659 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:04.766 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.664 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:04.772 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.679 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:04.787 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.711 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:04.779 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.714 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:04.782 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.717 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:04.824 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.718 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:04.826 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.728 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:04.796 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:53:51.733 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:04.840 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.498 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:13.605 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.546 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:13.653 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.562 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:13.670 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.591 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:13.658 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.595 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:13.663 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.608 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:13.676 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.613 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:13.721 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.616 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:13.724 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:00.630 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:13.738 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.560 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:18.667 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.563 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:18.671 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.578 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:18.686 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.610 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:18.678 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.614 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:18.681 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.625 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:18.693 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.633 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:18.741 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.634 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:18.742 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:05.645 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:18.753 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:11.625 INFO:tasks.ceph.osd.5:Stopped 2023-12-11T20:54:11.625 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-11T20:54:11.625 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:54:11.638 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:54:11.631+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86781) UID: 0 2023-12-11T20:54:11.638 INFO:tasks.ceph.osd.6.smithi133.stderr:2023-12-11T20:54:11.632+0000 b47e640 -1 osd.6 1075 *** Got signal Terminated *** 2023-12-11T20:54:17.727 INFO:tasks.ceph.osd.6:Stopped 2023-12-11T20:54:17.727 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-11T20:54:17.727 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:54:17.739 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:54:17.732+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86783) UID: 0 2023-12-11T20:54:17.739 INFO:tasks.ceph.osd.7.smithi133.stderr:2023-12-11T20:54:17.734+0000 b47e640 -1 osd.7 1075 *** Got signal Terminated *** 2023-12-11T20:54:20.605 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:33.712 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:20.607 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:33.715 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:20.627 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:33.735 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:20.657 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:33.725 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:20.658 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:33.726 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:20.660 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:33.768 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:20.663 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:33.771 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:20.681 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:33.789 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:20.683 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:33.751 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.617 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:38.724 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.621 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:38.729 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.633 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:38.741 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.670 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:38.778 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.673 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:38.781 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.680 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:38.748 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.681 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:38.749 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.687 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:38.795 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:25.694 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:38.762 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.655 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:53.762 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.656 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:53.764 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.677 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:53.785 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.708 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:53.776 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.708 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:53.816 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.709 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:53.817 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.710 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:53.778 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.724 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:53.792 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:40.728 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:53.836 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.673 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:58.780 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.674 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:58.782 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.688 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:38:58.796 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.720 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:58.788 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.722 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:58.830 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.722 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:58.790 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.725 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:58.833 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.734 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:38:58.802 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:54:45.738 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:38:58.846 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.711 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:13.819 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.716 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:13.824 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.743 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:13.851 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.764 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:13.832 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.768 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:13.836 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.769 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:13.876 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.771 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:13.879 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.789 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:13.897 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:00.793 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:13.861 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.688 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:18.795 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.734 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:18.801 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.736 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:18.804 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.737 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:18.845 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.752 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:18.820 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.753 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:18.861 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.792 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:18.899 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.793 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:18.901 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:05.807 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:18.914 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.815 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:33.923 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.820 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:33.928 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.841 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:33.949 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.867 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:33.935 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.869 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:33.977 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.871 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:33.939 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.875 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:33.983 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.888 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:33.956 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:20.896 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:34.004 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.809 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:38.917 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.812 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:38.920 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.813 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:38.920 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.827 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:38.935 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.859 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:38.927 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.862 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:38.930 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.869 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:38.977 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.876 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:38.944 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:25.881 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:38.989 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.845 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:53.953 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.849 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:53.957 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.866 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:53.974 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.896 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:53.964 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.899 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:54.007 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.905 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:54.013 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.912 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:53.980 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.929 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:54.036 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:40.936 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:54.004 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.871 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:58.978 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.873 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:58.981 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.886 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:39:58.994 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.913 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:58.981 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.919 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:58.987 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.936 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:39:59.004 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.945 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:59.053 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.947 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:59.055 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:55:45.957 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:39:59.065 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.858 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:13.966 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.906 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:13.974 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.907 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:13.975 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.911 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:14.019 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.934 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:14.002 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.939 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:14.046 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.967 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:14.075 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.970 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:14.078 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:00.988 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:14.096 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.922 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:19.029 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.923 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:19.031 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.934 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:19.042 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.970 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:19.038 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.971 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:19.039 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.976 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:19.084 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.977 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:19.085 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.984 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:19.051 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:05.988 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:19.096 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:20.976 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:34.083 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:20.978 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:34.086 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:20.994 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:34.102 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:21.022 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:34.090 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:21.024 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:34.092 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:21.025 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:34.133 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:21.029 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:34.136 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:21.040 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:34.108 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:21.045 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:34.153 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:25.991 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:39.098 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:25.993 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:39.101 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:26.006 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:39.114 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:26.038 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:39.106 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:26.041 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:39.109 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:26.042 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:39.150 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:26.044 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:39.152 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:26.053 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:39.121 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:26.058 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:39.166 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.126 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:54.233 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.129 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:54.237 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.142 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:54.210 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.146 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:54.253 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.146 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:54.213 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.175 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:54.283 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.178 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:54.286 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.190 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:54.258 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:41.224 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:54.332 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.078 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:59.186 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.079 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:59.187 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.091 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:40:59.199 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.124 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:59.191 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.125 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:59.193 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.126 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:59.234 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.129 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:59.237 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.139 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:40:59.207 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:56:46.143 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:40:59.251 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.120 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:14.228 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.121 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:14.229 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.135 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:14.243 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.162 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:14.230 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.165 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:14.233 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.181 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:14.249 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.190 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:14.298 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.193 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:14.301 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:01.210 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:14.318 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.421 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:15.529 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.422 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:15.530 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.435 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:15.543 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.472 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:15.540 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.473 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:15.541 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.477 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:15.585 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.479 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:15.587 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.486 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:15.554 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:02.492 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:15.600 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.130 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:19.238 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.134 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:19.242 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.147 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:19.254 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.182 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:19.289 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.182 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:19.249 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.184 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:19.253 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.188 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:19.296 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.197 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:19.265 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:06.199 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:19.307 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.177 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:34.285 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.182 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:34.290 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.196 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:34.304 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.223 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:34.291 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.226 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:34.294 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.230 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:34.338 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.234 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:34.342 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.243 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:34.311 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:21.250 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:34.358 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.187 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:39.294 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.191 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:39.299 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.206 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:39.313 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.234 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:39.302 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.236 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:39.304 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.236 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:39.344 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.240 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:39.348 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.249 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:39.317 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:26.251 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:39.359 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.237 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:54.344 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.239 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:54.347 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.260 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:54.368 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.283 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:54.351 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.286 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:54.354 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.295 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:54.403 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.298 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:54.406 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.301 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:54.369 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:41.315 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:54.423 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.245 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:59.352 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.248 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:59.356 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.261 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:41:59.369 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.294 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:59.362 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.295 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:59.363 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.297 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:59.405 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.300 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:59.408 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.308 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:41:59.376 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:57:46.313 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:41:59.421 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.259 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:14.367 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.310 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:14.418 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.350 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:14.458 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.357 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:14.425 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.359 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:14.427 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.361 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:14.469 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.364 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:14.472 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.374 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:14.442 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:01.382 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:14.490 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.303 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:19.411 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.305 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:19.413 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.318 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:19.426 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.350 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:19.418 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.353 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:19.421 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.353 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:19.461 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.355 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:19.463 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.364 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:19.432 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:06.366 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:19.474 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.368 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:34.476 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.385 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:34.493 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.405 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:34.513 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.424 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:34.492 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.425 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:34.493 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.436 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:34.544 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.439 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:34.547 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.439 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:34.507 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:21.456 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:34.564 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.362 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:39.470 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.364 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:39.472 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.377 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:39.485 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.408 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:39.476 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.411 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:39.479 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.422 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:39.491 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.428 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:39.536 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.430 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:39.538 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:26.442 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:39.550 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.405 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:54.513 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.409 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:54.517 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.426 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:54.534 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.457 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:54.525 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.458 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:54.526 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.462 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:54.569 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.464 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:54.572 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.474 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:54.542 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:41.479 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:54.587 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.416 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:59.524 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.420 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:59.528 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.432 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:42:59.540 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.467 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:59.575 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.467 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:59.535 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.468 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:59.537 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.470 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:59.578 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.479 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:42:59.547 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:58:46.480 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:42:59.588 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.460 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:43:14.568 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.465 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:43:14.573 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.489 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:43:14.596 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.513 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:43:14.581 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.514 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:43:14.582 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.521 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:43:14.629 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.524 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:43:14.632 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.530 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:43:14.598 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:01.541 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:43:14.649 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.482 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:43:19.589 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.483 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:43:19.591 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.494 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:43:19.602 86524== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.526 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:43:19.594 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.529 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:43:19.597 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.535 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:43:19.642 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.536 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:43:19.644 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.540 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:43:19.608 86167== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:06.548 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:43:19.656 86527== Warning: unimplemented fcntl command: 1036 2023-12-11T20:59:15.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-12-11T20:59:16.695 INFO:tasks.ceph.osd.7:Stopped 2023-12-11T20:59:16.695 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-11T20:59:16.696 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-11T20:59:16.696 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:59:16.718 INFO:tasks.ceph.mgr.y:Stopped 2023-12-11T20:59:16.719 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-11T20:59:16.719 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:59:16.720 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-12-11T20:59:16.720+0000 7fe46d0ed640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 86165) UID: 0 2023-12-11T20:59:16.748 INFO:tasks.ceph.mgr.x:Stopped 2023-12-11T20:59:16.748 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-11T20:59:16.749 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-11T20:59:16.749 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:59:16.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2023-12-11T20:59:16.763+0000 9f74640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86498) UID: 0 2023-12-11T20:59:16.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2023-12-11T20:59:16.765+0000 9f74640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-11T20:59:17.609 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:43:30.717 86524== 2023-12-11T20:59:17.609 INFO:tasks.ceph.mon.a.smithi086.stderr:==00:00:43:30.717 86524== Exit program on first error (--exit-on-first-error=yes) 2023-12-11T20:59:17.633 INFO:tasks.ceph.mon.a.smithi086.stderr:daemon-helper: command failed with exit status 42 2023-12-11T20:59:20.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-11T20:59:20.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-12-11T20:59:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-11T20:59:22.851 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-11T20:59:22.852 INFO:tasks.ceph.mon.a:Stopped 2023-12-11T20:59:22.853 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-11T20:59:22.853 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:59:22.867 INFO:tasks.ceph.mon.c.smithi086.stderr:2023-12-11T20:59:22.857+0000 9f74640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86500) UID: 0 2023-12-11T20:59:22.868 INFO:tasks.ceph.mon.c.smithi086.stderr:2023-12-11T20:59:22.859+0000 9f74640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-11T20:59:23.672 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:43:36.780 86527== 2023-12-11T20:59:23.672 INFO:tasks.ceph.mon.c.smithi086.stderr:==00:00:43:36.780 86527== Exit program on first error (--exit-on-first-error=yes) 2023-12-11T20:59:23.692 INFO:tasks.ceph.mon.c.smithi086.stderr:daemon-helper: command failed with exit status 42 2023-12-11T20:59:25.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-11T20:59:25.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-11T20:59:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-11T20:59:28.954 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-11T20:59:28.955 INFO:tasks.ceph.mon.c:Stopped 2023-12-11T20:59:28.955 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-11T20:59:28.955 INFO:teuthology.orchestra.run:waiting for 300 2023-12-11T20:59:28.970 INFO:tasks.ceph.mon.b.smithi133.stderr:2023-12-11T20:59:28.961+0000 9f74640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86163) UID: 0 2023-12-11T20:59:28.971 INFO:tasks.ceph.mon.b.smithi133.stderr:2023-12-11T20:59:28.962+0000 9f74640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-12-11T20:59:29.684 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:43:42.751 86167== 2023-12-11T20:59:29.684 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:43:42.751 86167== Exit program on first error (--exit-on-first-error=yes) 2023-12-11T20:59:29.703 INFO:tasks.ceph.mon.b.smithi133.stderr:daemon-helper: command failed with exit status 42 2023-12-11T20:59:30.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-11T20:59:30.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-11T20:59:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-11T20:59:35.057 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-11T20:59:35.057 INFO:tasks.ceph.mon.b:Stopped 2023-12-11T20:59:35.057 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-11T20:59:35.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-12-11T20:59:35.098 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:59:35.098 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-11T20:59:35.273 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:59:35.273 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-11T20:59:35.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:59:35.329 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-11T20:59:35.387 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:59:35.388 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-11T20:59:35.446 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:59:35.446 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-11T20:59:35.518 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:59:35.518 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-11T20:59:35.590 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:59:35.590 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-11T20:59:35.639 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:59:35.639 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-11T20:59:35.689 INFO:tasks.ceph:Archiving mon data... 2023-12-11T20:59:35.690 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/data/mon.a.tgz 2023-12-11T20:59:35.690 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-12-11T20:59:35.706 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.kEJksCX87r 2023-12-11T20:59:35.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kEJksCX87r 2023-12-11T20:59:35.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.kEJksCX87r 2023-12-11T20:59:35.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-11T20:59:35.954 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.kEJksCX87r is 427KB 2023-12-11T20:59:36.054 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.kEJksCX87r 2023-12-11T20:59:36.070 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/data/mon.c.tgz 2023-12-11T20:59:36.070 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-12-11T20:59:36.123 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.p7YJ3i13Hx 2023-12-11T20:59:36.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.p7YJ3i13Hx 2023-12-11T20:59:36.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.p7YJ3i13Hx 2023-12-11T20:59:36.345 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.p7YJ3i13Hx is 429KB 2023-12-11T20:59:36.442 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.p7YJ3i13Hx 2023-12-11T20:59:36.456 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/data/mon.b.tgz 2023-12-11T20:59:36.457 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2023-12-11T20:59:36.475 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.Q8iwcUXHAf 2023-12-11T20:59:36.475 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Q8iwcUXHAf 2023-12-11T20:59:36.641 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.Q8iwcUXHAf 2023-12-11T20:59:36.726 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.Q8iwcUXHAf is 429KB 2023-12-11T20:59:36.824 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.Q8iwcUXHAf 2023-12-11T20:59:36.838 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-11T20:59:36.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-11T20:59:36.840 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-11T20:59:37.165 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-11T20:59:37.429 INFO:tasks.ceph:Archiving crash dumps... 2023-12-11T20:59:37.431 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/remote/smithi086/crash 2023-12-11T20:59:37.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-11T20:59:37.460 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/remote/smithi133/crash 2023-12-11T20:59:37.461 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-11T20:59:37.491 INFO:tasks.ceph:Compressing logs... 2023-12-11T20:59:37.491 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-11T20:59:37.505 DEBUG:teuthology.orchestra.run.smithi133:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-11T20:59:37.534 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92980.log 2023-12-11T20:59:37.534 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90566.log 2023-12-11T20:59:37.534 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92601.log 2023-12-11T20:59:37.534 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91178.log 2023-12-11T20:59:37.534 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160051.log 2023-12-11T20:59:37.534 INFO:teuthology.orchestra.run.smithi086.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90566.log.gz 2023-12-11T20:59:37.534 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91178.log: /var/log/ceph/ceph-client.admin.160051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92371.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116097.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91178.log.gz 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160051.log.gz 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159841.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.116097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160261.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127539.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159799.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.159841.log.gz 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.127539.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.160261.log.gz 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92681.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90837.log 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92681.log: 88.8% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.127539.log.gz 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159799.log.gz 2023-12-11T20:59:37.535 INFO:teuthology.orchestra.run.smithi086.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.92371.log.gz 2023-12-11T20:59:37.536 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-12-11T20:59:37.536 INFO:teuthology.orchestra.run.smithi086.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.116097.log.gz 2023-12-11T20:59:37.536 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90837.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-12-11T20:59:37.549 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/osd.2.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90837.log.gz 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/osd.0.log: 64.0%/var/log/ceph/valgrind/osd.1.log: -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93201.log 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90829.log 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.92601.log.gz 2023-12-11T20:59:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92557.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144755.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92557.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90829.log.gz 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr: -- /var/log/ceph/ceph-client.admin.92511.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92681.log.gz 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.144755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93031.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91750.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93209.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.93031.log.gz 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128059.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93209.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91750.log.gz 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160135.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.128059.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.144755.log.gz 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93070.log 2023-12-11T20:59:37.551 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160135.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.160135.log.gz 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91454.log 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93070.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.128059.log.gz 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144753.log 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91454.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91454.log.gz 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93349.log 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.144753.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.93070.log.gz 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159757.log 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93349.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.93209.log.gz 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116095.log 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159757.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.144753.log.gz 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91173.log 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159757.log.gz 2023-12-11T20:59:37.552 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.116095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159715.log 2023-12-11T20:59:37.555 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91173.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91173.log.gz 2023-12-11T20:59:37.555 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93148.log 2023-12-11T20:59:37.555 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159715.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.159715.log.gz 2023-12-11T20:59:37.555 INFO:teuthology.orchestra.run.smithi086.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.116095.log.gz 2023-12-11T20:59:37.556 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160177.log 2023-12-11T20:59:37.556 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91920.log 2023-12-11T20:59:37.556 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160177.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.160177.log.gz 2023-12-11T20:59:37.556 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92766.log 2023-12-11T20:59:37.556 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91920.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91920.log.gz 2023-12-11T20:59:37.556 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-11T20:59:37.557 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92766.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.93148.log.gz 2023-12-11T20:59:37.558 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93098.log 2023-12-11T20:59:37.568 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-11T20:59:37.568 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-12-11T20:59:37.568 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-12-11T20:59:37.568 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/valgrind/osd.5.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86480.log 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr: 74.7%/var/log/ceph/valgrind/osd.4.log: -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86520.log 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.86480.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.86520.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86480.log.gz 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86520.log.gz 2023-12-11T20:59:37.569 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86571.log 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.86571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86611.log 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86400.log 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86571.log.gz 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.86400.log: /var/log/ceph/ceph-client.admin.86611.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86611.log.gz 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86400.log.gz 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-11T20:59:37.570 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86360.log 2023-12-11T20:59:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.audit.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92766.log.gz 2023-12-11T20:59:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93379.log 2023-12-11T20:59:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93098.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.93201.log.gz 2023-12-11T20:59:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160219.log 2023-12-11T20:59:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93379.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.93098.log.gz 2023-12-11T20:59:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159925.log 2023-12-11T20:59:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160219.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.160219.log.gz 2023-12-11T20:59:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92306.log 2023-12-11T20:59:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159925.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159925.log.gz 2023-12-11T20:59:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92639.log 2023-12-11T20:59:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92306.log: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-11T20:59:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92869.log 2023-12-11T20:59:37.577 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph.log: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-11T20:59:37.577 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86320.log 2023-12-11T20:59:37.577 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.86360.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.86360.log.gz 2023-12-11T20:59:37.577 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86440.log 2023-12-11T20:59:37.577 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.86320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86320.log.gz 2023-12-11T20:59:37.577 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.86440.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86440.log.gz 2023-12-11T20:59:37.577 INFO:teuthology.orchestra.run.smithi133.stderr: 88.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-11T20:59:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92639.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.93379.log.gz 2023-12-11T20:59:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86540.log 2023-12-11T20:59:37.585 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92869.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92980.log.gz 2023-12-11T20:59:37.585 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91163.log 2023-12-11T20:59:37.586 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.86540.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92639.log.gz 2023-12-11T20:59:37.586 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90524.log 2023-12-11T20:59:37.586 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91163.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.86540.log.gz 2023-12-11T20:59:37.588 INFO:teuthology.orchestra.run.smithi086.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91163.log.gz 2023-12-11T20:59:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91584.log 2023-12-11T20:59:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91504.log 2023-12-11T20:59:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91584.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90524.log.gz 2023-12-11T20:59:37.588 INFO:teuthology.orchestra.run.smithi086.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91584.log.gz 2023-12-11T20:59:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93458.log 2023-12-11T20:59:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160303.log 2023-12-11T20:59:37.596 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93458.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91504.log.gz 2023-12-11T20:59:37.596 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159589.log 2023-12-11T20:59:37.596 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160303.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.92869.log.gz 2023-12-11T20:59:37.596 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160009.log 2023-12-11T20:59:37.596 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4%/var/log/ceph/ceph-client.admin.159589.log: -- replaced with /var/log/ceph/ceph-client.admin.160303.log.gz 2023-12-11T20:59:37.596 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119141.log 2023-12-11T20:59:37.597 INFO:teuthology.orchestra.run.smithi086.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.159589.log.gz 2023-12-11T20:59:37.597 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159967.log 2023-12-11T20:59:37.597 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.119141.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160009.log.gz 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160387.log 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159967.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159967.log.gz 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91169.log 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160387.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.160387.log.gz 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90373.log 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.119141.log.gz 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91169.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90373.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91169.log.gz 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90373.log.gz 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92484.log 2023-12-11T20:59:37.598 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91870.log 2023-12-11T20:59:37.608 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92484.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.93349.log.gz 2023-12-11T20:59:37.608 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88729.log 2023-12-11T20:59:37.608 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91870.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91870.log.gz 2023-12-11T20:59:37.619 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92941.log 2023-12-11T20:59:37.619 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.88729.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88729.log.gz 2023-12-11T20:59:37.627 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90416.log 2023-12-11T20:59:37.629 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92941.log: 94.3% -- replaced with /var/log/ceph/ceph-client.admin.92557.log.gz 2023-12-11T20:59:37.629 INFO:teuthology.orchestra.run.smithi086.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.92306.log.gz 2023-12-11T20:59:37.631 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135973.log 2023-12-11T20:59:37.631 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93149.log 2023-12-11T20:59:37.631 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.135973.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90416.log.gz 2023-12-11T20:59:37.631 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91474.log 2023-12-11T20:59:37.633 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93149.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.93149.log.gz 2023-12-11T20:59:37.633 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91962.log 2023-12-11T20:59:37.636 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91474.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.135973.log.gz 2023-12-11T20:59:37.636 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135969.log 2023-12-11T20:59:37.636 INFO:teuthology.orchestra.run.smithi086.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91474.log.gz 2023-12-11T20:59:37.636 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135971.log 2023-12-11T20:59:37.637 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.135969.log: 92.8% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91962.log.gz 2023-12-11T20:59:37.637 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92941.log.gz 2023-12-11T20:59:37.637 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93405.log 2023-12-11T20:59:37.637 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.135971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93436.log 2023-12-11T20:59:37.639 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93405.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.135969.log.gz 2023-12-11T20:59:37.639 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91394.log 2023-12-11T20:59:37.639 INFO:teuthology.orchestra.run.smithi086.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.93458.log.gz 2023-12-11T20:59:37.640 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93436.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.135971.log.gz 2023-12-11T20:59:37.640 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92803.log 2023-12-11T20:59:37.640 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128061.log 2023-12-11T20:59:37.641 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92803.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91394.log.gz 2023-12-11T20:59:37.641 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87852.log 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.128061.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.93436.log.gz 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91175.log 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.87852.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.128061.log.gz 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87852.log.gz 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86842.log 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90839.log 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.86842.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91175.log.gz 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90839.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86842.log.gz 2023-12-11T20:59:37.646 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144754.log 2023-12-11T20:59:37.647 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.3.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90839.log.gz 2023-12-11T20:59:37.647 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92293.log 2023-12-11T20:59:37.648 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.144754.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92803.log.gz 2023-12-11T20:59:37.648 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127540.log 2023-12-11T20:59:37.649 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92293.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.144754.log.gz 2023-12-11T20:59:37.649 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160429.log 2023-12-11T20:59:37.652 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.127540.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.127540.log.gz 2023-12-11T20:59:37.653 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91534.log 2023-12-11T20:59:37.655 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160429.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.160429.log.gz 2023-12-11T20:59:37.656 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93197.log 2023-12-11T20:59:37.656 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91534.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91534.log.gz 2023-12-11T20:59:37.656 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90838.log 2023-12-11T20:59:37.656 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93197.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.93405.log.gz 2023-12-11T20:59:37.656 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93095.log 2023-12-11T20:59:37.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90838.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90838.log.gz 2023-12-11T20:59:37.660 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91622.log 2023-12-11T20:59:37.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93095.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.93197.log.gz 2023-12-11T20:59:37.660 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92892.log 2023-12-11T20:59:37.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91622.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91622.log.gz 2023-12-11T20:59:37.660 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92807.log 2023-12-11T20:59:37.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92892.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.93095.log.gz 2023-12-11T20:59:37.662 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91830.log 2023-12-11T20:59:37.664 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92807.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.92892.log.gz 2023-12-11T20:59:37.664 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92348.log 2023-12-11T20:59:37.664 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91830.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91830.log.gz 2023-12-11T20:59:37.665 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93207.log 2023-12-11T20:59:37.671 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92348.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.92807.log.gz 2023-12-11T20:59:37.671 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91174.log 2023-12-11T20:59:37.673 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93207.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.93207.log.gz 2023-12-11T20:59:37.673 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91172.log 2023-12-11T20:59:37.676 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91174.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91174.log.gz 2023-12-11T20:59:37.676 INFO:teuthology.orchestra.run.smithi086.stderr:gzip 92.1% -- replaced with /var/log/ceph/ceph-client.admin.92348.log.gz 2023-12-11T20:59:37.676 INFO:teuthology.orchestra.run.smithi086.stderr: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80689.log 2023-12-11T20:59:37.676 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91710.log 2023-12-11T20:59:37.676 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.tmp-client.admin.80689.log: 87.7% 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80689.log.gz 2023-12-11T20:59:37.676 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91172.log.gz 2023-12-11T20:59:37.676 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86992.log 2023-12-11T20:59:37.676 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127538.log 2023-12-11T20:59:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91710.log: /var/log/ceph/ceph-client.admin.86992.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91710.log.gz 2023-12-11T20:59:37.678 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86992.log.gz 2023-12-11T20:59:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-11T20:59:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90842.log 2023-12-11T20:59:37.680 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.127538.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.127538.log.gz 2023-12-11T20:59:37.681 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116094.log 2023-12-11T20:59:37.681 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90842.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90842.log.gz 2023-12-11T20:59:37.682 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128058.log 2023-12-11T20:59:37.683 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.116094.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.92293.log.gz 2023-12-11T20:59:37.684 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90072.log 2023-12-11T20:59:37.685 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.128058.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.116094.log.gz 2023-12-11T20:59:37.686 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92004.log 2023-12-11T20:59:37.686 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90072.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90072.log.gz 2023-12-11T20:59:37.690 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160345.log 2023-12-11T20:59:37.690 INFO:teuthology.orchestra.run.smithi086.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.128058.log.gz 2023-12-11T20:59:37.690 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90480.log 2023-12-11T20:59:37.690 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160345.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92004.log.gz 2023-12-11T20:59:37.690 INFO:teuthology.orchestra.run.smithi086.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.160345.log.gz 2023-12-11T20:59:37.690 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93205.log 2023-12-11T20:59:37.691 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92407.log 2023-12-11T20:59:37.691 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93205.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90480.log.gz 2023-12-11T20:59:37.691 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-11T20:59:37.707 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92407.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.93205.log.gz 2023-12-11T20:59:37.708 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90610.log 2023-12-11T20:59:37.708 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.0.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.92407.log.gz 2023-12-11T20:59:37.708 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90656.log 2023-12-11T20:59:37.708 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90610.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90610.log.gz 2023-12-11T20:59:37.708 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160512.log 2023-12-11T20:59:37.708 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90656.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90656.log.gz 2023-12-11T20:59:37.708 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119142.log 2023-12-11T20:59:37.708 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160512.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.160512.log.gz 2023-12-11T20:59:37.709 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159883.log 2023-12-11T20:59:37.709 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.119142.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119142.log.gz 2023-12-11T20:59:37.709 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92847.log 2023-12-11T20:59:37.709 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159883.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.159883.log.gz 2023-12-11T20:59:37.709 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93422.log 2023-12-11T20:59:37.709 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92847.log: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-11T20:59:37.713 INFO:teuthology.orchestra.run.smithi086.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92847.log.gz 2023-12-11T20:59:37.713 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91374.log 2023-12-11T20:59:37.725 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93422.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.93422.log.gz 2023-12-11T20:59:37.726 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92376.log 2023-12-11T20:59:37.726 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93399.log 2023-12-11T20:59:37.726 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91374.log: /var/log/ceph/ceph-client.admin.92376.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91374.log.gz 2023-12-11T20:59:37.727 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-11T20:59:37.731 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93399.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.93399.log.gz 2023-12-11T20:59:37.731 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159547.log 2023-12-11T20:59:37.744 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86869.log 2023-12-11T20:59:37.745 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159547.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159547.log.gz 2023-12-11T20:59:37.748 INFO:teuthology.orchestra.run.smithi086.stderr: 93.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-11T20:59:37.748 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-11T20:59:37.750 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.86869.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86869.log.gz 2023-12-11T20:59:37.750 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92583.log 2023-12-11T20:59:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159631.log 2023-12-11T20:59:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92583.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.92583.log.gz 2023-12-11T20:59:37.789 INFO:teuthology.orchestra.run.smithi086.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.92376.log.gz 2023-12-11T20:59:37.789 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92723.log 2023-12-11T20:59:37.790 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159631.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159631.log.gz 2023-12-11T20:59:37.790 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160472.log 2023-12-11T20:59:37.795 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92723.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92723.log.gz 2023-12-11T20:59:37.795 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90825.log 2023-12-11T20:59:37.797 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160472.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.160472.log.gz 2023-12-11T20:59:37.797 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160093.log 2023-12-11T20:59:37.798 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90825.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90825.log.gz 2023-12-11T20:59:37.798 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91790.log 2023-12-11T20:59:37.800 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.160093.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.160093.log.gz 2023-12-11T20:59:37.800 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93275.log 2023-12-11T20:59:37.801 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91790.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91790.log.gz 2023-12-11T20:59:37.801 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135972.log 2023-12-11T20:59:37.803 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.93275.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.93275.log.gz 2023-12-11T20:59:37.803 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159673.log 2023-12-11T20:59:37.807 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.135972.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.135972.log.gz 2023-12-11T20:59:37.807 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92488.log 2023-12-11T20:59:37.808 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.159673.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.159673.log.gz 2023-12-11T20:59:37.808 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91165.log 2023-12-11T20:59:37.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92488.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92488.log.gz 2023-12-11T20:59:37.814 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90826.log 2023-12-11T20:59:37.815 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.91165.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91165.log.gz 2023-12-11T20:59:37.816 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90831.log 2023-12-11T20:59:37.817 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90826.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90826.log.gz 2023-12-11T20:59:37.817 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92329.log 2023-12-11T20:59:37.818 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.90831.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90831.log.gz 2023-12-11T20:59:37.819 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-11T20:59:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.92329.log: 91.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135970.log 2023-12-11T20:59:37.827 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92329.log.gz 2023-12-11T20:59:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.135970.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.135970.log.gz 2023-12-11T20:59:37.887 INFO:teuthology.orchestra.run.smithi133.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-11T20:59:38.062 INFO:teuthology.orchestra.run.smithi086.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.92511.log.gz 2023-12-11T20:59:38.987 INFO:teuthology.orchestra.run.smithi086.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92484.log.gz 2023-12-11T20:59:39.779 INFO:teuthology.orchestra.run.smithi086.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-11T20:59:39.912 INFO:teuthology.orchestra.run.smithi133.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-11T20:59:40.896 INFO:teuthology.orchestra.run.smithi086.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-11T20:59:43.129 INFO:teuthology.orchestra.run.smithi086.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-11T20:59:43.214 INFO:teuthology.orchestra.run.smithi133.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-11T20:59:43.448 INFO:teuthology.orchestra.run.smithi133.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-11T20:59:43.567 INFO:teuthology.orchestra.run.smithi086.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-11T20:59:43.573 INFO:teuthology.orchestra.run.smithi133.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-11T20:59:43.965 INFO:teuthology.orchestra.run.smithi133.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-11T20:59:43.967 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-12-11T20:59:43.967 INFO:teuthology.orchestra.run.smithi133.stderr:real 0m6.417s 2023-12-11T20:59:43.967 INFO:teuthology.orchestra.run.smithi133.stderr:user 0m25.723s 2023-12-11T20:59:43.967 INFO:teuthology.orchestra.run.smithi133.stderr:sys 0m0.894s 2023-12-11T20:59:44.165 INFO:teuthology.orchestra.run.smithi086.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-11T20:59:44.392 INFO:teuthology.orchestra.run.smithi086.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-11T20:59:44.394 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-12-11T20:59:44.394 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m6.877s 2023-12-11T20:59:44.394 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m31.497s 2023-12-11T20:59:44.394 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m1.128s 2023-12-11T20:59:44.394 INFO:tasks.ceph:Archiving logs... 2023-12-11T20:59:44.395 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/remote/smithi086/log 2023-12-11T20:59:44.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-11T20:59:48.848 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/remote/smithi133/log 2023-12-11T20:59:48.849 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-11T20:59:52.660 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-11T20:59:52.662 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-11T20:59:52.671 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-11T20:59:52.671 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-11T20:59:52.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-11T20:59:52.674 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-11T20:59:52.758 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-11T20:59:52.767 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-11T20:59:52.767 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-11T20:59:52.769 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-11T20:59:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2023-12-11T20:59:52.785 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2023-12-11T20:59:52.788 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-11T20:59:52.788 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-12-11T20:59:52.788 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 +2471us[+2471us] +/- 110ms 2023-12-11T20:59:52.789 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -1150us[-1140us] +/- 51ms 2023-12-11T20:59:52.789 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +85us[ +85us] +/- 76ms 2023-12-11T20:59:52.789 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:59:52.790 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-11T20:59:52.790 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2023-12-11T20:59:52.790 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +2435us[+2435us] +/- 110ms 2023-12-11T20:59:52.790 INFO:teuthology.orchestra.run.smithi133.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -1203us[-1272us] +/- 51ms 2023-12-11T20:59:52.791 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 112 +96us[ +26us] +/- 75ms 2023-12-11T20:59:52.791 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-12-11T20:59:52.791 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-11T20:59:52.808 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-11T20:59:52.809 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-11T20:59:52.837 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-11T20:59:52.936 DEBUG:teuthology.orchestra.run.smithi133:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-11T20:59:53.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-11T20:59:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T20:59:53.068 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-11T20:59:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T20:59:53.101 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-11T20:59:53.110 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-11T20:59:53.119 INFO:teuthology.task.internal:Duration was 3753.940399 seconds 2023-12-11T20:59:53.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-11T20:59:53.129 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-11T20:59:53.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-11T20:59:53.132 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-11T20:59:53.172 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-11T20:59:53.184 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-11T20:59:53.566 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-11T20:59:53.566 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2023-12-11T20:59:53.566 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-11T20:59:53.626 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2023-12-11T20:59:53.626 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-11T20:59:53.661 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-11T20:59:53.661 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-11T20:59:53.669 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-11T20:59:53.730 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-11T20:59:53.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-11T20:59:53.739 DEBUG:teuthology.orchestra.run.smithi133:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-11T20:59:53.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-11T20:59:53.877 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-11T20:59:53.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-11T20:59:53.904 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-11T20:59:53.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-11T20:59:53.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-11T20:59:53.947 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-11T20:59:53.970 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2023-12-11T20:59:53.997 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2023-12-11T20:59:54.027 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-11T20:59:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T20:59:54.041 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-11T20:59:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-11T20:59:54.081 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-11T20:59:54.093 INFO:teuthology.task.internal:Transferring archived files... 2023-12-11T20:59:54.094 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/remote/smithi086 2023-12-11T20:59:54.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-11T20:59:54.153 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265/remote/smithi133 2023-12-11T20:59:54.153 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-11T20:59:54.192 INFO:teuthology.task.internal:Removing archive directory... 2023-12-11T20:59:54.193 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-11T20:59:54.195 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-11T20:59:54.241 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-11T20:59:54.254 INFO:teuthology.task.internal:Not uploading archives. 2023-12-11T20:59:54.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-11T20:59:54.264 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-11T20:59:54.264 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-11T20:59:54.267 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-11T20:59:54.279 INFO:teuthology.orchestra.run.smithi086.stdout: 265773 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 11 20:59 /home/ubuntu/cephtest 2023-12-11T20:59:54.300 INFO:teuthology.orchestra.run.smithi133.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 11 20:59 /home/ubuntu/cephtest 2023-12-11T20:59:54.301 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-11T20:59:54.311 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-11T20:59:54.352 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-11T20:59:54.352 INFO:teuthology.nuke:Checking targets against current locks 2023-12-11T20:59:54.379 DEBUG:teuthology.nuke:shortname: smithi086 2023-12-11T20:59:54.379 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-11T20:59:54.404 DEBUG:teuthology.nuke:shortname: smithi133 2023-12-11T20:59:54.405 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-11T20:59:54.408 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-11 19:49:01.164038', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMOJn5q+AoDi076ehgEb2UB7o6YSW8kCF1AYQaNGSo6OsPXHsWuu+qyowrTGgX9c9Y5SWcZZUcD7KSBtoPIZFXI='} 2023-12-11T20:59:54.434 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_16:41:51-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7487265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-11 19:49:01.165693', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvBdaZakCDp41o3yJ8GUZkZbVVpPi2BSMpZMLhkk3GfTViBdqH1gJ1b4iArrB2Z2NsODYFNDvs6n8UV3I9Ceps='} 2023-12-11T20:59:54.463 INFO:teuthology.orchestra.console.smithi086:Power off 2023-12-11T20:59:54.463 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-11T20:59:54.489 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2023-12-11T20:59:54.489 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-11T20:59:54.511 INFO:teuthology.orchestra.console.smithi133:Power off 2023-12-11T20:59:54.512 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-11T20:59:54.523 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2023-12-11T20:59:54.540 DEBUG:teuthology.orchestra.console.smithi133:power off output: Chassis Power Control: Down/Off 2023-12-11T20:59:54.541 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-11T20:59:54.567 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-12-11T20:59:58.524 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-11T20:59:58.568 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-11T20:59:58.636 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2023-12-11T20:59:58.678 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-12-11T21:00:02.638 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-11T21:00:02.679 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-11T21:00:02.748 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2023-12-11T21:00:02.791 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-12-11T21:00:02.849 INFO:teuthology.orchestra.console.smithi086:Power off completed 2023-12-11T21:00:06.793 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-11T21:00:06.905 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2023-12-11T21:00:07.006 INFO:teuthology.orchestra.console.smithi133:Power off completed 2023-12-11T21:00:07.110 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3753.9403986930847 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-12-11T21:00:07.111 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-11T21:00:07.184 INFO:teuthology.run:FAIL